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|. Introduction 


Recent years have seen the emergence of design-thinking and maker learning 
initiatives across formal and informal learning environments. A variety of offerings, 
from museum and afterschool programs, to STEM challenges and entrepreneurship 
courses, create opportunities for young people to develop skills relevant to today’s 
workplace. Design-based learning emphasizes the importance of creative problem- 
solving, taking perspective and empathizing, identifying problems worth solving, and 
iterating on solutions. Yet despite high relevance to 21° century workforce demands, 
opportunities for young people to engage long-term in design-based experiences 
remain limited. 


With the support of HP and Microsoft, Digital Promise Global's Learning Studios 
program brought enhanced design-based learning opportunities to teachers and 
students in 60 schools in the US and abroad. The Learning Studios program provided 
advanced technology, design-based learning resources, and an online teacher 
community to participating sites, and the first implementation spanned the 2016-17 
school year. To capture and share outcomes and insights from the Learning Studio 
implementation project with the larger community, an exploratory research 
component was included as well. 


This report presents the key outcomes of that research, and is organized in four main 
sections. To set the research in context, details on project resources and 
implementation are described in the remainder of this section. In the second section, 
we explain how the research was organized and the ways that data were gathered 
and analyzed. Section three presents key thematic findings across survey, interview 
and focus group data. In the fourth and final section of the report we reflect on 
opportunities for additional research and share recommendations for future 
implementations of Learning Studios and other design-based education programs. 
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1.1 About Learning Studios 


Learning Studios is a design-based learning initiative directed by Digital Promise 
Global as part of HP and Microsoft's Reinvent the Classroom initiative. The goal of the 
project is to create new opportunities for student-centered, experiential learning by 
equipping schools with advanced technologies for creation and collaboration. A total 
of sixty school sites spanning 11 countries participated in the program between May 
2016 and May 2017, as illustrated in Figure 1. Each site received a technology and 
resources package that included a 3D printer, a desktop computer with integrated 3D 
scanner and interactive touchmat, and notebook computers, as well as additional 
technologies and materials. A full list of the Learning Studio package appears in 
Figure 2. 


Country (# sites) 
e Australia (4) 


2 e Canada (5) 


x ° Finland (1) 
“a e Germany (1) 
e Italy (1) 


e New Zealand (1) 
e Norway (1) 


co) e Spain (1) 
%» e Sweden (1) 


e United Kingdom (3) 
e United States (41) 


Figure 1: Learning Studios Implementation Countries and Sites 


e Sprout by HP 
e 10 HP x360 310 G2 Convertible Notebooks 
e Dremel idea Builder 3D printer 
e HP Printer with Pagewide Technology : 
e Large format presentation display | 
e Notebook storage unit. a 
e Supplemental kit: DREMEL 
o 4 Makey Makey invention kits 
(donated by Joylabz) 
o iclass pack of Model Magic (donated by Crayola) 
o 2 additional spools of PLA filament (donated by 
Dremel) 


o 1 Cardboard Construction Kit (tape measure, utility 
shears, fasteners, etc) 


Oo 1 Take-It-Apart Kit (pliers, screwdrivers, etc) 


Figure 2: Learning Studios Technology and Equipment 
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In addition to providing technology resources to participating schools, Digital 
Promise Global offered a range of instructional and professional development 
supports to teachers. An online Teacher's Guide, together with a collection of ten 
project guides for students, were available for educators to draw upon at their 
discretion. Four of the projects were recommended in a sequence between 
September and November 2016 for teachers to prepare students for the culminating 
Global Goals, Local Solutions project. From January to May 2017, an additional three 
project guides were published leading up to the final Play to Learn project. Sample 
project guides from both phases are included in Appendix A. 


Teachers also had access to an online community in which they could connect with 
other Learning Studio educators to share and collaborate on new ideas, discover 
resources and project ideas, share classroom stories, and get support and tips from 
Digital Promise Global instructional coaches on ways to use related technology tools 
in their classrooms. Teachers also had access to a webinar series conducted by 
program staff, and one-day of professional development on the technology, 
provided in person by Educational Collaborators. 


1.2 Implementation Models 


At each site, a “Lead Educator” served as the primary point of contact with Digital 
Promise Global. Schools were encouraged to invite additional teachers to use the 
equipment, projects, and online supports as well. While a single teacher engaged 
with the program at most sites, as many as seven teachers participated at some sites. 
In terms of timeframe, the original Learning Studio implementation spanned May 
through December 2016. In response to the project team’s conversations with 
participating teachers during the fall of 2016, the project was extended through the 
end of the U.S. school year. The original set of project guides remained available to 
teachers. Digital Promise Global added a second culminating challenge and related 
supplemental projects focused on game design and titled “Play to Learn”. From 
January to May 2017, at each teacher's discretion, students continued to work in the 
Learning Studio on the projects provided by the program, or on other projects, using 
the equipment and resources provided, and took part in the newly added Play to 
Learn Challenge. 


For several reasons, implementation of the Learning Studio program was markedly 
different across sites. Sites were selected from a range of locations within the United 
States and abroad to create a global community of participants, and to create 
opportunities for teachers and students to interact with peers around the world. In 
addition, sites were selected across grade levels, from elementary through high 
school. Instructional context was another distinguishing characteristic. Sites were 
diverse in this respect, ranging from afterschool programs, to use with existing 
curricula such as Project Lead the Way, to weekly maker sessions. Moreover, teachers 
brought diverse backgrounds in design-based teaching and learning to their 
participation as facilitators of student engagement. Some teachers had extensive 
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experience with design-based teaching, and were comfortable with students taking 
the lead on their design projects and with the technology. In contrast, others were 
relatively new to the technology as well as to instructional approaches In which the 
teacher serves as facilitator. 


An additional factor contributing to the variety of implementation models is that the 
program intentionally gave teachers wide latitude to choose which projects to carry 
out with their students. Learning Studios imposed minimal requirements on 
participating teachers and students, asking only that they complete the Global Goals, 
Local Solutions Challenge project between November and December 2016. 
Conceived as a culminating project in which students would draw upon the 
knowledge and skills they had built through preceding projects, the Global Goals, 
Local Solutions Challenge prompted students to collaboratively design and produce 
a local solution to a United Nations Sustainable Development Goal.’ Finally, the 
extent to which sites took advantage of the 2017 extension of the program varied, 
and in some cases a new cohort of students came on board in January of 2017. 


Taken together, the contextual differences and the variations in implementation 
across sites resulted in substantial differences across students’ experiences with the 
Learning Studios. This diversity was a deliberate characteristic of the Learning Studio 
program, and we intentionally gave precedence to implementation values over 
research design considerations, which would have limited differences across site 
characteristics and required similar project work for all students. In the following 
section describing the research design, we elaborate further on the implications of 
the Learning Studio implementation diversity for interpretation of results. 


2. Research Design and Data Collection 


Research on the Learning Studio initiative was carried out by Designs for Learning in 
collaboration with Digital Promise Global, representing a hybrid third-party approach. 
Designs for Learning maintained an independent role in data collection and analysis, 
while also working cooperatively with the Digital Promise Global team to inform the 
impact goals for the project and to interpret results. 


The purpose of the Learning Studios research was to understand how teachers and 
students made use of the advanced technologies provided by the project, and to 
document the areas of growth experienced by students and teachers—from 
technology skills to design-based thinking and related competencies. We also sought 
to understand the effects of implementation context on technology use and learning. 
Finally, we wanted to capture insights related to challenges and best practices that 
could inform the broader field. 


1 
http://www.un.org/sustainabledevelopment/sustainable-development-goals/ 
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As noted in the previous section, the grassroots nature of the Learning Studio 
program yielded substantial variation across sites in terms of context and 
implementation strategies. We expected that students and teachers would benefit 
from participation in the program; however, we also knew that benefits were likely to 
vary with contextual factors such as grade level, curriculum constraints, and student 
exposure. 


Another factor informing our choice of research design was the relative lack of prior 
studies on outcomes associated with design-based learning and maker learning 
initiatives. Because our research would contribute to a nascent body of knowledge 
about the ways that design-based learning experiences benefit students, we were in 
the position of anticipating outcomes rather than relying on previous studies to know 
what to look for. A related consequence of the current state of research is that the 
research instruments themselves—survey questions and protocols—had to be 
developed from scratch to capture changes in these outcomes, largely without the 
benefit of drawing on previously validated surveys. 


For these reasons, we designed the research strategy to be exploratory in nature— 
balancing our expectations for specific outcomes with opportunities for 
unanticipated benefits to emerge. As such, we adopted a mixed-methods approach, 
integrating both quantitative and open-ended qualitative prompts on surveys to all 
participants, and conducting interviews and focus groups with teachers and students 
at several sites. We hoped that taking this approach would enable us to document 
the impacts of engagement with the Learning Studios program, as well as to identify 
trends for further exploration and replication, notwithstanding the research 
challenges implicit in a highly diverse implementation sample. 


2.1 Literature Review 


To inform our methods and guide our expectations for teacher and student 
outcomes, we carried out a literature review to identify and summarize existing 
frameworks and learning outcomes associated with making and design-centered 
learning environments. We also looked to the literature to identify existing methods 
and measures for gathering evidence of these outcomes among learners. Our 
techniques for conducting the review included online searches of published 
manuscripts and white papers, as well as email exchanges, phone interviews, and in- 
person meetings with active researchers. While we identified several relevant 
frameworks, our review found few empirical studies, and little in the way of quasi- 
experimental designs. Figure 3 presents a summary of the findings that directly 
informed our design. 
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Agency by Design. Developed by a team of researchers at Harvard Project Zero, the Agency by Design framework defines 


maker empowerment as “a sensitivity to the designed dimension of objects and systems, along with the inclination 
and capacity to shape one’s world through building, tinkering, re/designing or hacking” (Agency by Design, 2015). 
Key dimensions are sensitivity, which includes looking closely, exploring complexity, and finding opportunity; 
inclination and motivation to act; and the capacity and ability to be effective. 


Learning Dimension Framework. Researchers at the Exploratorium have iterated on a “Learning Dimension Framework" 


through their investigations of museum visitors’ engagement with tinkering and making in informal spaces. The four 


dimensions described in Petrich, Wilkinson & Bevan (2013) are engagement, intentionality, innovation and solidarity. 


Next Generation Science Standards (NGSS) practices. Creativity Labs, in collaboration with the Maker Education Initiative, 


focused their research on NGSS practices, and found that the practices most applied in maker settings included: 
developing and using models; planning and carrying out investigations; and analyzing and interpreting data 
(Peppler, Maltese, Keune, Change & Regalla, 2015). 


Democratization of Making. Researchers from the FabLab group at Stanford University, led by Paolo Blikstein, have focused 


on the intersection of digital fabrication and the democratization of making. Research describes their innovative 


approaches to identification and assessment of learner outcomes associated with maker spaces (Blikstein, Fields, 


Kabayadondo, & Martin, 2017). 


Maker Mindset. AnnMarie Thomas, a leader in the maker education field and former director of the Maker Education 


Initiative, describes the maker mindset as a set of maker qualities including curiosity, playfulness, openness to risk, 
responsibility, persistence, resourcefulness, generosity in sharing, and optimism (Thomas, 2014). 


Figure 3: Learning Frameworks Associated with Design-based Learning and Making 


2.2 Outcomes Focus 


Informed by the review of the literature as well as input from project staff and 
collaborators, we decided to focus our research on four main outcomes. For each 
outcome, we created a set of specific indicators based on the literature, and then 


developed instruments to capture student and teacher activity and progress on these 


indicators. 


4. 
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Engagement and persistence 

Indicators: 

e Students are excited about coming to school; 

e Students are eager to visit the Learning Studio; 

e Adiverse range of students demonstrates their enthusiasm for the Learning 
Studio; 

e Students apply skills and perspectives from the Learning Studio to other 
aspects of life both in and out of school; and, 

e Students seek deeper understanding and knowledge beyond the minimum 
requirements. 


Agency and ownership of learning 

Indicators: 

e Setting personal learning goals; 

e Seeking and responding to feedback; 

e Persisting to achieve goals despite setbacks; 
e Taking intellectual risks; and, 
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e Personalizing projects or assignments. 


3. Empathy, collaboration and communication 
Indicators: 
e Awareness of the importance of taking perspective and understanding 
where others are coming from; 
e Ability to communicate effectively in a group work environment; and, 
e Ability to work effectively in a team, such as recognizing strengths that 
others bring, compromising and working through disagreements. 


4. Design thinking and problem-solving 
Indicators: 
e Recognizing that the material world is designed; 
e Ability to identify problems to solve; 
e Ability to take perspective on one’s own creation and those of others; 
e Demonstrating variation of efforts; and, 
e Ability to recognize failure and iteration as a regular part of the design 
process. 


2.3 Research Design 


To achieve our goal of documenting specific gains over time while also positioning 
the study to identify unanticipated outcomes, we crafted a research design that 
combined quasi-experimental and descriptive approaches. This approach would also 
enable us to capture participant input on program effectiveness and benefits. We 
gathered data from teachers and students using three methods: surveys, individual 
interviews, and focus groups, using the timeline shown in Figure 4. 


Teachers were invited to complete four surveys: a baseline ("PRE") administered in 
the spring of 2016 as the program was launched and teachers were invited to join the 
program; a mid-course survey ("MID") at the start of the 2016-17 school year; and a 
post survey ("POST") administered at the end of the first Challenge project, in 
December of 2016. In May of 2017, teachers were invited to complete a brief follow- 
up ("FOL") survey. All teachers who completed the baseline survey were also invited 
to a brief phone interview at the start of the 2016 school year, to ask about their 
implementation plans and progress to date. 


Students participating in the Learning Studio were invited through their teachers to 
complete a pre-survey as close as possible to their first visit to the studio, and a post- 
survey following the Challenge project in December of 2016. In May of 2017, 
teachers distributed a follow-up survey to students at the end of the second 
Challenge project. 
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In addition, a qualitative design was included to provide richer insights into change 
over time, and in particular to shed light on contextual factors related to Learning 
Studio implementation and outcomes. Ten schools were selected to serve as focal 
study sites. At each site, the lead teacher was interviewed individually via phone or 
Skype up to five times from October 2016 to June 2017. To gather student input, we 
also conducted virtual focus groups of up to five students at a time, from October to 
December 2016. Different students participated in each of the focus groups per site. 


April-Aug 2016 Sept 2016 Oct 2016 Nov 2016 Dec 2016 


Jan-Apr 2017 


Teachers 


Students 


Teacher Surveys 


Teacher 
Interviews 


Student Surveys 


Teacher 


Student Focus = 2c a 
Groups 


Figure 4: Research Design 


2.4 Teacher and Student Participants 


In this section we provide an overview of the teachers and students who participated 
in the research component of the Learning Studios project. Details of the survey 
instruments and interview protocols used to gather data are described in the section 
that follows. 


2.4.1 Teachers 


A total of 95 teachers from the 60 Learning Studio sites completed one or more of 
the surveys. In most cases, a single "Lead Educator" per site engaged with the 
program, although at some sites multiple teachers participated. Table 1 shows 
response numbers for each survey, as well as the number of teachers completing 
pairs of surveys. Overall, 62 took the pre-survey, 40 responded to the mid-survey, 30 
took the post-survey, and 27 completed the follow-up survey. Of teacher 
respondents, 20 completed both the pre and post surveys, and 18 completed both 
the mid and post surveys. Among teachers who completed the follow-up survey, 19, 
17, and 15 had responded to the pre-, mid- and post-survey, respectively. Seventeen 
teachers were interviewed in September about their plans for the Learning Studio; 
and ten teachers took part in the focal site interviews. 
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May 2017 


Table 1: Number of Teacher Responses for Pre, Mid, Post and Follow-up Surveys 


Totals: 62 40 30 27 
PRE MID POST FOL Overlap 
20 


Teachers were more or less evenly split across gender, and were fairly experienced as 
educators. About half (53%) of teachers taking the pre-survey were male; 45% were 
female. At the pre-survey, teachers reported an average of 13 years (sd=8.7) 
experience as educators, 6.5 years (sd=6.1) at their current organization, and 5.8 
years (sd = 5.5) in their current role. Regarding prior making experience, two-thirds of 
teachers reported having led makerspaces or programs in the past. As detailed in 
Figures 5 and 6, the majority (89%) of teachers identified with Caucasian/White 
ethnicity; and 60% have earned graduate degrees. 


Aboriginal 18% | 1 Associate/Advanced diploma 1.8% | al 
African/ African American/ Black 3.6% | 2 Bachelor 20.0% | 11 
Asian/ Pacific Islander / Pacific 1.8% | 1 Bachelor Honours/Honours 10.9% l 6 
peoples 

Masters 58.2% eet 32 
Caucasian/ European / White 89.1% || 49 

Doctorate 1.8% | 1 
Latino(-a) / Hispanic 9.1% | Si 
Middle Eastern 1.8% | 1 
Figure 5: Teacher Ethnicity Figure 6: Teacher Education 


At the pre-survey, teachers also provided information about the sites at which the 
Learning Studios were implemented. The majority (86%) were public/government 
schools; about 13% were private or independent schools and 2% were charter 
schools. Sites were roughly split evenly across suburban (44%) and urban (40%) 
settings, with 16% in rural locations. Participating schools enrolled from 120 to 2,030 
students. Among US-based sites, on average 45% of students were eligible for 
free/reduced price lunch. 
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2.4.2 Students 


A total of 1,276 students representing 52 teachers’ classrooms completed at least one 
survey. As shown in Table 2, of the overall sample, 401 students, representing 21 
teachers, responded to both pre and post surveys. Figure 7 presents key 
demographic information on students for both samples. At the follow-up, a total of 
297 students’ responses were obtained from 11 teachers’ classrooms. Across all three 
time-points, 55 students completed both the pre-survey and follow-up survey; 50 
completed the post- and follow-up surveys. In light of the attrition from pre-survey 
to follow-up, and due to the higher variance in program implementation in 2017, the 
focus of the quantitative results is on the primary implementation period, from 
September through December of 2016. When relevant, follow-up data are presented 
alongside the pre-post analyses. 


Table 2: Number of Student Responses for Pre, Post and Follow-up Surveys 
Totals: 1054 622 297 


PRE POST FOL Overlap 
7 / 401 


155 
150 


v 


NOTE: In this report, the label “Full Sample” refers to responses from the 1,276 
students who completed at least one survey; “Final Sample” refers to the 401 
students who completed both surveys. 


¥ 78% 
64% 
60% 55% 55% 
45% 45% 

40% 36% 

22% 
20% 
0% 

US 


Male Female 


International English-speaking elem (4-5) middle (6-8) high (9-13) 
country 


Full Sample MFinal Sample 


Figure 7: Student Demographics for Full and Final Samples 
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2.5 Data Collection: Surveys and Protocols 


Figure 4 shows when each data collection method was implemented in the context 
of the Learning Studio program. All participants were invited to complete surveys at 
key points in the project timeline; at a small number of sites, teachers and students 
were invited to a series of interviews and focus groups. The instruments for data 
collection were all designed to tap into the key outcomes guiding the research, and 
to capture information regarding context of implementation. Descriptions of each 
survey and protocol with teachers and students are provided in this section, and the 
complete instruments are available from the project team on request. 


2.5.1 Teacher Surveys 


Pre-survey. The pre-survey for teachers, administered in the spring of 2016, was 
organized in several sections. The first targeted demographics and professional 
experience, as well as basic demographics related to school setting and enrolled 
students. The second section probed teachers’ teaching style and instructional 
strategies in their regular classroom, and their planned strategies for the Learning 
Studio. In the third section, teachers shared their comfort levels with a variety of 
program-related activities and processes. The next section posed open-ended 
questions about teachers’ prior experience both making personally and facilitating 
students’ making activities, followed by a set of open-ended prompts about how 
teachers planned to use the Learning Studios equipment personally and with 
students. In a block of brief open-ended questions, teachers were asked to 
summarize the project in their own words, and to describe expectations such as 
anticipated challenges, struggles, personal growth, and ideal outcomes. A section 
focused on implementation asked teachers to describe the location and expectations 
for student involvement with the Learning Studio. Finally, teachers responded to 
open-ended prompts regarding their hopes for student learning and growth in the 
Learning Studio. 


Mid-survey. Teachers completed the brief mid-survey in September 2016. Items 
asked about level of practice to date with the Learning Studio equipment and 
familiarity with the project guides provided by program staff. A set of open-ended 
questions asked teachers to share ways they anticipated students demonstrating 
engagement, agency, initiative and collaboration in the Learning Studio. They also 
indicated whether students with learning differences (e.g., diagnosed learning or 
attention issues) would take part in the Learning Studio, and to describe expectations 
for those students’ experience vis-a-vis agency and engagement in the Learning 
Studio. 


Post-survey. The post survey, administered in December after completion of the first 
Challenge project (Global Goals, Local Solutions), repeated items posed in the pre- 
survey and mid-survey, with the exception of the demographic and site questions. 
Wording was modified to be retrospective rather than prospective. For example, the 
question asking “Please describe some of the ways you anticipate students 
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demonstrating engagement with the Learning Studio program” was changed for the 
post-survey to “Please describe some of the ways you saw students demonstrating 
engagement with the Learning Studio program." 


Follow-up survey. The follow-up survey was administered in May 2017, after 
completion of the Play to Learn Challenge, the second culminating project. This 
short survey included a subset of selected-response items posed in the previous 
surveys regarding teachers’ personal and instructional comfort with technology and 
making processes, and the frequency of Learning Studio activities such as public 
presentations of student projects and student access outside of regular school hours. 
The survey asked teachers about student participation overall and the projects 
completed, and included additional items regarding teachers’ perspectives on 
student experience of the Learning Studio. The last set of questions asked teachers to 
reflect on the extent to which participation in the Learning Studio had an impact on 
them personally and on their students. 


2.5.2 Student Surveys 


Pre-survey. Students completed the pre-survey in school at the start of their Learning 
Studio experience. The first blocks of items asked students to select their site and 
teacher, and provide basic demographic data on gender and ethnicity. The second 
section probed students’ identity as makers, asking to rate whether they see 
themselves as a designer or maker, and to describe previous experiences making, 
including what kinds of making they do, and with whom. In the third section, 
students rated their prior exposure to and current comfort levels with the Learning 
Studios equipment. Students were also prompted to share their ideas for how they 
might use the equipment. The next set of questions asked students to anticipate how 
much time they'd spend in the Learning Studio, as well as what they think they might 
learn, and what the experience will be like. Three items, adapted from Blikstein and 
colleagues (2017) were then posed to tap students’ design thinking knowledge. The 
following set of items echoed the questions posed to teachers regarding comfort 
with making processes. 


Post-survey. The post-survey repeated the same questions as the pre-survey, 
additionally asking which of the projects they completed, and what were the most 
important things they learned in the Learning Studio. An additional block of open- 
ended questions asked students to share their experience in the final project, 
including what they learned, what they found most challenging, most surprising, and 
what they enjoyed the most. Finally, students responded to Likert-style questions 
asking how often they took actions related to agency, collaboration, and persistence. 


Follow-up survey. The follow-up survey repeated a subset of selected-response 
questions from the pre and post surveys, including students’ identity as a maker and 
their comfort levels with technology and various making processes and skills. As on 
the post-survey, the follow-up survey also asked students how often they engaged In 
behaviors reflective of agency, collaboration and persistence in the Learning Studio. 
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2.5.5 Focal Site Protocols 


Of the ten focal sites, nine were located in the United States and one was in Australia. 
At each focal site, one teacher took part in up to five semi-structured interviews 
between October 2016 and June 2017. The interview protocol included questions 
about how the project was going, which projects students tried and how they went, 
and what kinds of growth and learning the teacher was observing. Teachers were 
asked to share how students who might not thrive in traditional classroom 
environments were doing in the Learning Studio. Finally, teachers were asked to 
describe their current goals for student learning, whether they had changed since the 
previous conversation, and what their overall goals for the program were. The 
interviews lasted between 20 minutes and an hour, with substantial probing of 
responses by the interviewer. Interviews were audio recorded and transcribed for 
analysis. 


The student focus group protocol guided conversations with small groups of 
students at each site, via Skype or other similar video conferencing platforms. The 
interviewer started off by asking students to share how it had been going so far in the 
Learning Studio, what it was like working in the Learning Studio, and what projects 
they were doing in the Learning Studio. Students were asked to describe what they 
had learned so far, what had been most challenging, and what surprised them the 
most. Whenever possible, students were prompted by the interviewer to provide 
concrete, specific examples, and the interview incorporated frequent probes for 
students to elaborate upon and clarify their input. In the later focus groups, students 
were asked to describe the final project as well. Student focus groups were audio- 
recorded for later analysis. 


3. Results 


As a whole, the surveys, interviews and focus groups generated data that helped us 
understand the extent to which the intended outcomes of the Learning Studio 
program had been attained. These data also provided insights into participants’ 
perspectives on the value of the program, and enabled us to make connections 
between aspects of the implementation context and observed outcomes. 


This section presents the results thematically, starting with information about the 
contexts of implementation and moving to students’ and teachers’ comfort with the 
specific technologies. Following these foundational results, we explore participants’ 
growth and perspective within each of the four key outcome areas. We then present 
findings regarding the Learning Studio experience for students who have struggled in 
traditional classroom settings. Finally, we share results of analyses examining the 
intersection of instructional context and student outcomes. 


Data from teacher and student surveys as well as interviews and focus groups are 
integrated within each section. Where multiple surveys posed the same question, we 
tested growth in two ways: by comparing responses on early (pre, mid) surveys to 
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responses on later (post, follow-up) surveys; and by asking teachers and students to 
reflect on their experience in the Learning Studio. Unless otherwise specified, to test 
hypotheses regarding gains from early to later points in the program implementation, 
one-tailed paired-sample t-tests were performed. It is important to note that the 
small sample size for complete data sets, which was especially true for teachers, 
meant that reaching statistical significance required large gains. As this was an 
exploratory study, we opted to examine trends as well as statistically significant 
outcomes. 


3.1 Background & Implementation Context 


Because learning does not happen in a vacuum, context is as critical as the 
technologies and resources. For that reason, we focused a portion of the research on 
understanding several aspects of the context of implementation for the Learning 
Studios. Through open-ended questions at the very start of the program, we asked 
teachers to share their prior experiences with designing and making, whether 
personally or as educators. Other questions asked teachers to indicate their 
pedagogical approaches and priorities. We also placed importance on knowing how 
often students visited the Learning Studio, and what kinds of projects they worked on 
while there. Implementation model was another aspect of context we considered. 


Implementation Characteristics. As expected, we found tremendous diversity in the 
settings in which students engaged with the program and the ways that the Learning 
Studio was being implemented at each school. From grade level to socioeconomic 
markers to teachers’ own prior experience with making and technology, participating 
classrooms differed substantially. In some schools the primary implementation was 
as an afterschool program. Others implemented the Learning Studio in the context of 
an existing design course, such as Project Lead the Way, whereas some teachers 
allocated an hour a week within a math or science class. In the majority of cases, 
students’ participation was required as part of their school curriculum, however at 
several sites use of the Learning Studio was voluntary or part of an afterschool club. 


Educator Backgrounds. We also observed a great deal of variety in the backgrounds 
that teachers brought to the Learning Studio. An early analysis was carried out to 
qualitatively group teachers into categories based on their personal and professional 
experiences with maker and design-centered learning. The entire set of responses, 
both quantitative and qualitative, were used to holistically generate categories. This 
process yielded four groups of teachers, in which no attempt was made to create 
evenly sized groups. A sampling of responses from three of the survey questions are 
included in Appendix B for each group. 


Group 1: Beginner (19 teachers) 
e enthusiasm and buy-in for the Learning Studio project 
e little background in making or design thinking principles or strategies 
e did not identify design thinking as an area of growth for their students in the 
Learning Studio 
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e tended to focus on technology skills 
Group 2: Emerging (7 teachers) 
@ some personal experience with making 
e little/no experience facilitating students to make 
e did not express awareness of design thinking principles or strategies 
Group 3: Intermediate (14 teachers) 
@ personal and professional experiences with making 
e experience facilitating student making projects, including robotics clubs and 
other initiatives 
e design thinking did not figure prominently in responses 
Group 4: Advanced (12 teachers) 
e evidenced knowledge and experience of the design process 
e situated making within a larger culture of pedagogy related to project-based 
learning 


Pedagogy and Instructional Strategies. We further explored context by asking 
teachers about their pedagogical approach, and whether their approaches in their 
classrooms differed from their teaching in the Learning Studio. Overall, teachers 
described fairly progressive approaches, and a desire to engage students actively and 
in authentic challenges. One source of evidence for this finding was the descriptions 
of their teaching style that teachers provided on the pre-survey. Among the common 
themes that emerged in those responses were: 


Emphasis on collaboration (mentioned 12 times): 

e “| have a collaborative classroom where every student gets the opportunity to 
participate. | use small group instruction and stations on a daily basis. | also use 
technology on a regular basis.” 

e “lama teacher that encourages collaboration and cooperation. | encourage 
my students to think out of the box. | encourage the use of effective 
technology to facilitate, enhance, support and enrich their learning." 

e “Hands-on, lots of inquiry-based, small group instruction and collaborative, 
student-led learning /problem solving.” 


Emphasis on problem-solving (mentioned 11 times): 

e ‘| try to find out with student a precise maths topic by reasoning and problem 
solving. | do not share with students a new theory from zero, but | want them 
to reach the focus point by themselves.” 

e “My role as an educator is a facilitator of learning. | try to implement as many 
opportunities for personalized learning including facets of voice, choice, pace, 
path, and place In my teaching. | rely on an environment that is very much 
student-led as well as try to foster a collaborative environment around 
problem solving.” 
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e “| like students who are able to find solutions to problems using all the 
resources (technological or not) that they have in the class. | like to do a short 
introduction before starting the class to explain the goals that we have during 
the session and help the students to learn how to find themselves the 
solutions. Sometimes you have to explain concepts, but they should learn by 
doing." 


Hands-on and interactive approaches: 

e “| like students to be able to be hands-on. | find that students, especially 
middle school students, learn best by doing. This is especially true when it's 
something they are interested in. | try to limit the lectures to providing them 
with the background knowledge they need, and then give them the 
opportunity to explore and learn." 

e “Hands-on, lots of inquiry-based, small group instruction and collaborative, 
student-led learning /problem solving.” 

e “lama very interactive teacher! | try to avoid sit and get teaching by creating 
activities that are engaging and meaningful for students. Technology plays an 
important role in this level of engagement and | enjoy allowing my students to 
extend their thinking through the use of technology. In addition to working 
with students in a whole group setting, | also pull students for small group 
instruction in order to provide more individualized instruction. All of these 
components fit well within the PBL framework, which is an integral part of 
teaching and learning.” 


Teachers’ endorsement of facilitative approaches to instruction also came through 
when we asked them to report how often they use specific instructional strategies in 
their regular classroom, and in the Learning Studio. Table 3 presents their pre-post 
responses, ordered by the proportion of teachers who reported using the strategy or 
not in each setting. Generally, instructional strategy uses were ranked similarly by 
teachers for the classroom and Learning Studio. However, teachers tended to rank 
overseeing collaborative activities more highly for the Learning Studio than in their 
classroom, where the top-ranked strategy pertained to facilitating student 
discussions. Notably, across the board, participating teachers selected instructional 
strategies consistent with design-based learning, for instance with tests and direct 
instruction ranked toward the bottom of the list. 
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Table 3: Teachers’ Rankings (1 through 12) of Instructional Strategy Use in Their Classroom and the Learning Studio 


Demonstrate uses of technology 1 2 1 
Facilitate student discussions of approaches to solving 5 2 3 2 
problems, explanations of their thinking, or open-ended 

questions 

Demonstrate a concept using graphics, computer, projector 1 3 4 5 
Set up and monitor or supervise cooperative or collaborative 4 4 1 4 
learning activities 

Observe or monitor student-led whole-group class discussions vi 6 6 5 
or demonstrations 

Provide individual or small group tutoring as needed during 3 5 7 6 
individual seatwork or small group activities 

Demonstrate a concept using physical artifacts or models 6 7 B} 7 
Provide remedial or enriching instruction to a pull-out group 10 8 8 8 
while the rest of the class works on assignments 

Lecture, perhaps occasionally soliciting student input or using 8 9 9 9 
board/overhead to highlight a key term of present an outline 

Administer a test or quiz 9 10 10 10 
Work on administrative tasks, such as grading, while students 12 12 12 11 
work on assignments individually 

Lead students in recitation, drills, or question-and-answer 11 1. tt 12 
sessions 


Because we were curious to see whether teachers’ instructional strategies would 
change over the course of their Learning Studio implementation, we asked for these 
rankings at the post-survey as well. We also posed the open-ended question about 
teaching style again at the post-survey. As Table 4 shows, we found that teachers’ 
ratings remained fairly constant from pre to post. Similarly, we did not observe shifts 
in teachers’ descriptions of their teaching style over the course of the program. 
However, one interesting finding was the increase in order of importance of the 
strategy “Facilitate student discussions of approaches to solving problems, 
explanations of their thinking, or open-ended questions” in teachers’ regular 
classrooms. Among teachers completing both pre and post surveys, on average this 
strategy ranked 5" at pre, and moved up to 2 at post. With respect to instructional 
strategies specifically anticipated or used in the Learning Studio, notable changes 
from pre to post include a drop in importance, from 1“ to 4"" place, of “Set up and 
monitor or supervise cooperative or collaborative learning activities”. While an early 
trend, and subject to confirmation in future studies, this outcome might suggest that 
teachers discovered a need to provide more support around the technology than 
originally anticipated. 


In interviews, several teachers spoke to the ways that educating in a Learning Studio 
environment called for a different approach to instruction—one in which they did not 
need to Know all the answers, and in which students could also feel free to try new 
things. As one high school teacher reflected when asked how to bring other teachers 
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onboard, “Accepting that mindset, and telling everyone from the beginning that we 
don't expect staff to know everything. It sounds really simple but when you 
acknowledge that with the kids it starts the relationship off differently than ‘| know 
everything and you don't’.” 


With regard to pedagogy and instructional approaches, there was some evidence that 
the early nature of instruction in design thinking means that teachers often have to 
rely on their own observations to identify typical learning challenges, and to craft 
effective instructional strategies in response. For example, an elementary school 
teacher reflected on how she discovered that students’ understanding of 3D 
dimensional shapes was a key determinant in whether they could engage 
successfully in 3D modeling. “I realized that before we can talk about 3D modeling 
we really needed to talk about three-dimensional shapes-- which are [state] 
standards all the way through. Even the little kids need to understand that objects 
look different from different perspectives. | give the example in my class of a teddy 
bear. If you were to draw a teddy bear with shapes, you'd probably use a circle for the 
head, maybe an oval for a body, some little circle ears, maybe a triangle nose, right? 
You can imagine what those shapes look like. But then if | tell you to draw the teddy 
bear looking at it from the top down, it becomes harder -- because the top of the 
teddy bear doesn't look like a teddy bear. (...) | guarantee whoever's struggling with 
that are the same kids who will struggle with the 3D modeling. | think [those 
foundational skills] are the lowest level scaffolding before students can be successful 
in the 3D modeling piece.” 


Student Exposure to the Learning Studio. The quantity and quality of students’ 
exposure to the Learning Studio was another key aspect of context to consider. To 
assess exposure, we asked teachers how often students visited the Learning Studio 
and how long they spent there. We also asked which of the project guides provided 
by the program staff teachers opted to implement with their students. Figures 8 and 
9 present data showing weekly exposure to the Learning Studio. For the majority (18) 
of teachers who responded, students typically made up to 3.5 visits to the Learning 
Studio each week, and spent up to 4 hours there weekly. However, a third (6) of the 
teachers indicated that their students visited about once a week. In other words, over 
a three-month period, that group of students would have spent about 15 hours total 
in the Learning Studio—a relatively short amount of time to expect dramatic learning 
and growth. 
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Student LS Visits per Week Student LS Hours per Week 
(#Teachers reporting) (#Teachers reporting) 
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Figures 8 and 9: Teacher-reported Student Weekly VISITS and HOURS Spent in Learning Studio 


In many cases, students chose to access the Learning Studio beyond the times 
allotted in class. As one teacher shared, “Yesterday at lunch | had seventy kids in here! 
They love their project and want to work on it.” And another, “Even outside of their 
class time, students come in to work on their independent project. Probably more 
than twenty students every morning doing some kind of independent project.” A 
middle-school teacher shared, “They like the Learning Studio because it gives them 
more freedom. (...) They now know that they can come in there and not only have 
the equipment but also the ability to do something. | think the freedom that they get 
really makes their eyes light up and makes them want to come.” 


Because teachers were at liberty to choose which projects, if any, to implement as 
part of the Learning Studio, exposure to the Projects is another important 
consideration in making sense of the students’ responses. As can be seen in Figure 
10, an important finding was that teachers implemented only a few of the Learning 
Studio projects. Although the Global Goals, Local Solutions project was strongly 
encouraged, only about two-thirds of teachers implemented it with their students. 
Specifically, at the post survey, twenty-eight teachers responded to the question 
about Projects completed. On average, their students completed two of the projects 
provided by the Learning Studio program, with a range from zero to five projects 
completed. Because different samples of teachers responded to the post and follow- 
up surveys, the number of times a project was completed vary. 
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Learning Studio Projects Implemented 


BPost BFollowUp 


Hack n Jam 

interactive Classroom® 
Five Chairs* 

Honey, | Shrunk the Engine 
Toy Workshop* 

Moving Pictures 

Paper Boat* 

Custom Cookie Cutter 


Global Goals, Local Solutions 


i 


Parts, Purposes, Complexities* 


Play to Learn Challenge 


Figure 10: Learning Studio Projects Implemented by Participating Teachers *'s denotes “projects of the week" that teachers were 
encouraged to use in the fall leading up to Global Goals, Local Solutions Challenge. 


3.2 Technology Use, Comfort & Related Skills 


In this section, we look at the extent to which teachers and students used the 
advanced technologies in the Learning Studio, how their comfort with these 
technologies changed over time, and the skills they developed in using them over the 
course of the program. 


Practice and Comfort with Learning Studios Equipment. Not surprisingly, one of the 
impacts of the Learning Studio was to increase access for teachers and students to 
advanced technologies. From September (mid) through December (post), we saw 
significant gains in teachers’ practice using the Sprout and 3D printer technology 
(Table 4A). The same was true on average for students from pre to post surveys 
(Table 4B). Another view of the same data shows shifts from pre to post, and from pre 
to follow-up surveys, of the distribution of students across the response options. 
Figures 11 through 14 reveal substantial shifts from “not yet” or “not sure what this is” 
responses into the “once or twice” through “many times” responses. The proportion 
of students who had used the Sprout at least once increased from 23% to 65% from 
pre to post. For the 3D printer, the increase was from 29% to 58%. At the same time, 
at post and at follow-up, a high proportion of students shared that they had not yet 
used these technologies (39%-51%). 


One possible explanation for this finding, which emerged in interviews with teachers, 
is that the ratio of technology to students made it challenging to give everyone 
access. Teachers and students noted that time was also a factor, especially for the 3D 
printer, which can require many hours to print a single design. For this reason, 
students had to rotate access, and it is possible that for students working in a group, 
they were not the person designated to work directly with the printer. At one high 
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school, students explained that because so demand was so great for the 3D printer, a 
few students were responsible for all printing. In that scenario, these results may 
under-represent the extent to which students had the opportunity to carry out their 
design work using the Learning Studio resources. Another explanation for why such a 
large proportion of students did not report using the Sprout or 3D printer is that while 
the technology was a valuable resource for teachers, there were many options for 
introducing students to design thinking and making. For instance, the Parts, Purposes 
& Complexities project, based on the thinking routine developed by Agency by 
Design (2015), that several teachers implemented with students did not require the 
use of technology. As such, those students may have still benefited from the 
program, even if they had limited hands-on experience with the technologies. 


Table 4A: Teachers’ Use of Learning Studio Equipment+ 


MID POST 

Mean Sd Mean sd t 
Sprout 2.86 1.283 Si 1.066 (20) SSS Ole 
3D printer 2.86 1352 4.05 1.071 t(20)3.627 
Laptop 2.64 1.329 3.55 eval t(21)=3.177 


+5-point scale from 1 (“None yet") to 5 (“A lot") 


Table 4B: Students’ Use of Learning Studio Equipment+ 


PRE POST FOL n=401 n=43 

Mean | Sd Mean sd Mean sd PRE-POST PRE-POST-FOL 
Sprout 2.06 | 935 | 3.02 | 1168 | 3.07 | 1.486 t(400)=14.092°— F(1.85, 77.82) = 16.051 
3D printer | 2.37 | er SO GVM] 3.00 | 1.27 t(400)=10.593"— F(1.56, 65.45) = 12.316 
+ 5-point response scale: 1 (“Not sure what this is") — 2 (“Not yet’) — 3 (“Just once or twice’) — 4 ("A few times") — 5 (“Many 


times"); For pre and post columns, descriptive statistics are for the set of complete pre-post responses, for the follow-up 
column, descriptive statistics are for the set of complete pre-fol responses 


40% 


MManytimes MA fewtimes Once or twice Notyet M#Notsure 


Figure 11: How often did students use the Sprout (PRE-POST) n=401 
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Figure 12: How often did students use the Sprout (PRE-FOL) n=55 


MManytimes MAfewtimes MOnceortwice MNotyet MNotsure 


Figure 13: How often did students use the 3D printer? (PRE-POST) n=401 


MManytimes MAfewtimes MOnceortwice M™Notyet M[Notsure 


Figure 14: How often did students use the 3D printer? (PRE-FOL) n=55 


In addition to understanding how often teachers and students used the technology, 
we were also interested in how their comfort changed over time. We found 
substantial increases for teachers and students. For students, this comfort extended 
beyond using the technology to higher levels of comfort teaching others to use it 
and even for fixing technical problems that might arise. 


Tables 5 and 6 summarize the relevant survey responses for teachers and students. 
Over the first six months of their participation in the Learning Studio, teachers’ 
personal comfort with the technology increased significantly. To examine trends in 
students’ levels of comfort, we first removed those who reported not having used the 
technology yet from analyses. Among students who had used the Sprout or 3D 
printer at least once by the post or follow-up survey, we found significant gains for all 
areas. (Appendix B presents results for the full sample, without removing students 
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who didn't report use of the Sprout or 3D printer). Among the 24 students who 
completed all three surveys and used the technology at least once, comfort levels 
increased from the start to the end of the school year, and in particular from the post 
to the follow-up, suggesting that additional exposure and perhaps additional time 
matters for building comfort and knowledge of the technology. Despite small sample 
sizes, these results still registered as statistically significant. 


Mean sd Mean sd Mean sd 
FOL MID POST MID-POST (n=20) | PRE-POST-FOL (n=13) 
3.15 555 Sprout Pomieiom) 3.35 | 587 ijesver™ leveosoonaises” 
3.46 .660 3D printer 250m tooo Ws40N 598 t(19)=4.414° ‘| F(1.46,17.478) = 11.531" 
FOL PRE POST PRE-POST PRE-POST-FOL 
Sprout** n=246 n=24 
- Use Cavan eeSon| sisi) 982 t(245)=6.941 
3.96 1.083 2.88 | 1513 | 3.50 | 1319 F(2,46)=5.781 
- Teach DAN | 1.349) |RSS eer t(245)=7.721 
3.58 1.530 2.04 | 1.429 | 2.67 | 1.308 F(2,46)=10.191 
- Fix 2.37 | 1274) 2.78 | 1184 | +(245)=4.863"" 
3.37 1.469 1.96 | 1122 | 2.37 | 1.209 F(2,46)=12.570 
Mean sd Mean sd Mean sd PRE-POST PRE-POST-FOL 
FOL 3D printer*™* PRE POST n=232 n=23 
- Use 3.19 | 1437 | 3.88 | 1.012 | ¢t(231)=7.385 ~~ 
3.65 1.112 2.83 | 1586 | 3.48 | 1.238 F(2,44)=3.431 
- Teach 241 | 1342 | 3.21 | 1.239] ¢t(231)=8.395 
3.22 1.565 SOomisers) 2.70 | 1.521 F(2,44)=4.156° 
- Fix 226 | 1211 | 2.68 | 1215 | +t(231)=4.912°" 
3.09 1.535 EIO7me lsc7e 217 E54 F(2,44)=5.237 
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Mean sd Mean sd Mean sd 
PRE POST FOL PRE-POST PRE-POST-FOL 
Sprout’ ~~ n=246 n=24 
- Use 3.17 | 1.456] 3.81 | .982 t(245)=6.941 
Boomieoiom) 3.50 | 1319 3.96 1.0 F(2,46)=5.781 
83 
- Teach Pee etm) 3.15 | 1.171 t(245)=7.721 
2.04 | 1.429 | 2.67 | 1.308 3.58 15 F(2,46)=10.191 
30 
- Fix Cmalieem| 2.78 | 1.184 t(245)=4.863 
oom lee 2370922099) 3.37 1.4 F(2,46)=12.570 
69 
Mean sd Mean sd Mean sd PRE-POST PRE-POST-FOL 
3D n=232 
; +44 
printer PRE POST FOL n=23 
- Use 3.19 | 1.437 | 3.88 | 1.012 t(231)=7.385 
2.83 |1586 | 3.48 | 1238 | 3.65 1.11 F(2,44)=3.431 
2 
- Teach Pein ier) 3.21 | 1.239 t(231)=8.395 
DOOM MEsoce 270 tsa) 3.22 15 F(2,44)=4.156° 
65 
- Fix Acme) 2.68 | 1.215 t(231)=4.912 
oe soe 217 (549) 3.09 15 F(2,44)=5.237 
35 


+5 -point response scale: 1 “Not at all comfortable"; 2 “Not very comfortable"; 3 “Somewhat comfortable”; 4 “Comfortable”; 5 
“totally comfortable”. 

“This analysis includes only those students who reported having used the Sprout or 3D Printer at least once by the time of the 
POST or FOL survey. 

*"The first row for each result shows descriptive and test statistics for students with complete pre-post response sets; the 
second row shows statistics for students with complete pre-post-fol response sets. 

*“9<.001, “p<.01, ‘p<.05 


Regarding teachers’ comfort facilitating students’ use of the technology, from pre- 


survey to post-survey, teachers indicated increases. At the follow-up, these trends 
remained statistically significant, as shown In Table 7. 
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Table 7: Teachers’ Comfort Facilitating Students’ Use of Learning Studios Equipment+ 


MID POST FOL n=20, 19 n=13 

Mean sd Mean sd Mean sd MID-POST MID-POST-FOL 
Sprout 2.15 1.226 | 3.20 616 3.46 519 t(19)=4.273 | F(1.163,13.955)=15.857- 
3D printer | 2.63 1.165 3.32 749 3.46 660 t(18)=2.822° F(1.77,21.238)=3.973 


° For pre and post columns, descriptive statistics are for the set of complete pre-post responses; for the follow- 
up column, descriptive statistics are for the set of complete pre-fol responses. 
p<.001, p<.01,, p<.05 


Teachers’ own comfort with the technology, and perhaps their comfort in not having 
mastered the technology, seemed to influence students’ growth and comfort. “In the 
beginning, | felt like | needed to really know how to do all the different things [the 
Sprout and 3D printer] could do. And | felt like | wasn’t able to make refined models. 
But | found that as long as | didn't have that expectation of myself, the kids weren't 
blocked by that barrier.” Discovering techniques for using the technology effectively 
also made a difference, “Someone finally told me that the reason [3D printing] is so 
hard is that your model is not touching the bottom in all four corners. Once | started 
doing that | had no problems printing. So just those little tips and tricks were helpful. 
And as we discovered those things through trial and error, | think it made all the kids 
feel more comfortable.” 


The focal site teachers also reflected a variety of approaches to troubleshooting the 
technology, and these differences appeared tied to prior experience leading maker- 
type spaces. One experienced teacher explained, “| don’t teach 3D printing and | 
don't teach how to use the Sprout. They figure it out. | really like that component. | 
feel like they are fed so much from us and this is a place where they can take 
initiative.” In contrast, in one interview another high school teacher shared that 
students hadn't accessed the printer for a month, because it wasn't working and she 
hadn't had time to fix it. These patterns may be exacerbated in schools with limited 
resources, where the consequences are greater if valuable technology breaks; 
teachers in those situations may be reluctant to give students free reign to figure out 
solutions given the sense of risk entailed. 


Changes to Technology Skills. Beyond growing in their comfort with the technology, 
teachers and students also developed skills related to using the equipment and 
software. In later sections we describe how the Learning Studio experience drew 
upon a broad array of competencies for teachers and students; in this section we 
focus closely on specific technology-related skills. 


The surveys included questions about two specific technology skills related to 


making: recording and editing an audio or video recording, and creating a digital 3D 
model of an object. Results on these skills were mixed. For teachers, significant 
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improvements were found for creating a 3D model of an object, but not for working 
with audio or video files. Among students, we looked at middle school and high 
school students’ results separately and found distinct differences, shown in Table 9. 
Whereas middle school students reported a marginally significant increase from pre 
to post for working with audio and video files, the increase was not as notable for 
high schoolers. In contrast, comfort creating a digital 3D model remained nearly 
constant from pre to post for middle schools students, while significant gains were 
found among high school students. For students who completed all three surveys, as 
shown in Table 8, trends were both positive and significant despite a relatively small 
sample size. As we found with students’ comfort using the technology, these skills 
also appear to benefit from extended access and engagement with the Learning 
Studio, as demonstrated by continuously increased values from September 2016, to 
December 2016, to May 2017. One explanation for this finding is that more of the 
provided projects explicitly called for 3D modeling than for media production, which 


was often featured as an optional documentation element of the projects. 


Table 8: Overall and Grade-Level Pre-Post Changes in Student Comfort with Technology Skills 


PRE POST FOL 
[Tech Skill Mean sd Mean sd Mean sd PRE-POST PRE-POST-FOL 
Recording and editing an 
audio or video recording 
All (n=305) QGe 1.004 2.84 1.031 t(303)=1.790° 
All (n=38) 2.66 994 2.79 1.044 3.24 751 F(2,74)=6.421 
Middle School (n=178) 2.67 1.035 2.81 1.067 t(177)=1.591" 
High School (n=126) 2.79 .958 2.87 .980 t(125)= .729 (ns) 
Creating a digital 3D 
model of an object 
All (n=305) 2.70 1.059 2.79 1.059 t(303)=1.479" 
All (n=38) 225 || Les7 2.63 1.125 Dep || Ose F(2,74)=6.990 
Middle School (n=178) 2.97 1.008 2.99 977 t(177)=.347 (ns) 
High School (n=126) Se) 1.018 Aroill 1.108 t(125)=1,878° 


* Response scale ranged from 1 (“Low”) to 4 (“High) 


* 0<.05; ** p<.01; * p<.10 


As with technology use, we were curious whether teachers’ comfort facilitating 


students’ skill growth increased over the duration of the program. We found 


marginally significant increases for working with audio and video files, t(14)=1.382, 
p=0.094, and significant gains facilitating students to create a digital 3D model of an 
object, t(14)=2.982, p=0.005. At the follow-up, this latter trend remained statistically 
significant. The follow-up survey also included opportunities for teachers to indicate 
their agreement with a variety of statements about the program and their students’ 
experience of it. Despite the inclusion of a “neutral” option on the 5-point scale, 100% 
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of teachers agreed that their students learned valuable skills through their 
participation in the Learning Studio, with 63% expressing strong agreement. Among 
teachers who implemented the Play to Learn challenge with their students, strong 
agreement with this statement was at 88%. 


The conclusion that students learned more about technology was further evidenced 
in the open-ended responses on the post surveys. We asked students to briefly 
describe what they had learned in the Learning Studio. Students shared a variety of 
areas in which they had experienced growth. More than two in five (42%) cited 
technology as a learning area. Their responses ran the gamut from basic skills—e.g., 
“How to create my own things with a 3D printer’—to awareness of more robust 
growth—e.g,, “I've learned how to work better in groups, how to use modern 
technology like the sprout computer, solve problems without the teachers, work in 
different ways.” A number of responses focused on coding and programming, 
without reference to the equipment, e.g., "| have mostly learned a lot about 
computer programming in this class using the computers and laptops,” and “| learned 
how codes work and different types of codes. | also learn more about programs like 
Python, Java, C++." Projects related to coding were among the additional project 
guides that accompanied the Play to Learn Challenge. 


Gains in technology skills were also described by teachers in open-ended survey 
responses inquiring into outcomes of the Learning Studio program. One middle- 
school teacher wrote, “So far, | would say the biggest success for my students is that 
they have become far more proficient in 3D design with Auto-CAD-type software. 
Almost all of my 180 students in the last trimester were able to successfully 
design/build different models that followed specific guidelines. None of them were 
able to do that prior to receiving teaching and practice that involved the Learning 
Studio.” Another teacher cited “greater knowledge, engagement and usage of 
technological devices, improved digital literacy,” among the key outcomes. In 
interview conversations with teachers, we also found examples of ways that working 
with the Learning Studio technologies prompted students to build their skills in 
related standards. “One thing | noticed is strengthened by the modeling is 
measurement. | think that their concept of what's a millimeter, how big something is, 
has definitely been strengthened by the 3D modeling. | have one group that’s making 
a solar-powered Car ...The first [part] they made...they ended up having to print a 
bigger one. They figured out that 3cm is small! Now they can visualize that, which 
makes it easier for things like word problems or some of those higher-order 
Common Core math problems." 


3.3 Identity and Confidence as Designers and Makers 


This section presents the third and final area of inquiry that sets the backdrop for 
exploring the key outcome themes. We looked into the ways that the Learning Studio 
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experience helped teachers and students grow in their self-confidence and identity 
as designers and makers. As with the other topics, we approached this question 
through a variety of survey questions and interview prompts. Overall, we found 
evidence that while many students developed a stronger sense of themselves as 
makers and designers over the course of the project, this was not a universal 
outcome. This result differed by grade level, with older students reporting relatively 
higher levels of growth than younger students, with relatively more younger students 
nevertheless identifying as designers or makers. 


Changes in Students’ Sense of Identity as Designers and Makers. From the surveys, 
we learned that the change in sense of self as designer or maker from pre to post 
survey was positive and significant on average for high-school students, but not for 
middle or elementary students, though the latter showed a positive trend. It is 
noteworthy, given prior research on declines in motivation and self-efficacy as 
students progress from elementary through to middle and high school, that at the 
post survey, nearly two-thirds (64%) of elementary students considered themselves 
makers whereas 39% of middle schoolers and 53% of high schoolers felt the same 
(Table 9). Moreover, as shown in Table 11, among high school students who 
completed all three surveys, the average rating for this question at the follow-up was 
higher than at pre or post, though not statistically significantly so. 


A closer look at students’ pre-post responses to the question, “Do you see yourself as 
a designer or maker" reveals that at the pre-survey, 85 students chose the highest 
point on the scale and therefore could not show higher responses at the post survey. 
However, of the remaining 316 students, 36% gave a higher answer at post than at 
pre; 64% gave an equal or lower response. To further explore this finding, we carried 
out the analysis again, including only students who reported making at least 10 visits 
to the Learning Studio between pre and post surveys. Results were identical to the 
full sample, with 36% indicating growth in their identification as a designer or maker. 
Responding to a related question that appeared only on the follow-up survey, four in 
five students (80%) felt that taking part in the Learning Studio was helpful for seeing 
themselves more as a maker or designer, as shown in Figure 16. 
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Table 9: Proportions of Students Who Consider Themselves to be Makers or Designers 


Grade 
Band 
Elem 
(grades No 
4-5; Maybe 
n=G) Yes 
Middle 
(grades No 
6-8; Maybe 
n=198) _ 
High 
(grades No 
9-13; Maybe 
n=134) Ves 


PRE 


9% 
33% 
58% 


28% 
29% 
42% 


20% 
37% 
43% 


POST 


12% 
25% 
64% 


27% 
34% 
39% 


17% 
30% 
53% 


HSPRE Ea 
HS POST a 
MSPRE ee | 
MS POST Saas 
Elem PRE I peas 
Elem POST iy Ea 
0% 20% 40% 60% 80% 100% 


Wiyes Maybe [MNo 


Figure 15: Percentages of Students Who Consider Themselves to be Makers 
or Designers, by Grade Level 


Table 10: Changes in Students’ Maker Identity+ 


PRE POST FOL 
Are you a Maker or Mean sd Mea sd Mean sd PRE-POST PRE-POST-FOL 
Designer? n 
All S37 Alsi 3.44 1.176 t(400)=1.279 (ns) 
Elementary School (69)** | 3.75 1.077 3.88 1.078 t(68)=.869 (ns) 
Middle School (n=198)** | 3.24 | 1246 | 319 | 1.211 t(197)=-.563 (ns) 
High School (n=134) rove eoosm| 3158 | 1.085 t(133)=2.634 
High School (n=35) 3.40 dey 3.54 1.010 SAL 1.073 ae aati 
ns. 
* Response scale: 1 (Not at all) - 2 (Not so much) — 3 (Maybe) — 4(I guess so) — 5(Yes). 


** For elementary and middle school students, size of complete pre-post-fol data sets (n=4 for each) was too 


small to analyze. 
p<.01 
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Did taking part in the Learning Studio help you see yourself 
more as a maker or designer? 


™@ Definitely Not. ®Notsomuch M™®Notsure M@lIthinkso & Definitely Yes 


Figure 16: Learning Studio and Making Identity (FOL) 


In addition to asking students to select a response option to the question about their 
identity as makers and designers, we asked them to elaborate on their response 
choice. To qualitatively explore shifts in identity, we compared the pre and post 
responses of the 381 students who completed both surveys. About one in six 
students (16%) showed some increase in identification, with several describing 
marked changes. Examples are shown in Table 11. Our analyses also revealed a 
tendency for some students to interpret the question in the context of a career 
preference, or in relation to fashion design rather than more broadly. For example, “I 
do like to make things but that's not what | would like to do as a career” or "A 
designer is someone who creates many different pieces, whether it involves 
designing homes or designing fashion.” Because of this disconnect between the 
intent of the question and its interpretation, results may not be fully reflective of the 
ways that students developed greater confidence in their capacity to design, and a 
stronger sense of self as designer or maker. 
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Table 11: Pre-Post Responses to Why Students Consider Themselves to be a Maker or Designer — Examples of strongest gains 


PRE POST 

e I'm not much of a maker because | don't really > | e | said maybe because | feel like | have the 
invent anything or | don't often have ideas that capability and potential to become a 
work out. designer/maker in the future. 

e | can see myself as a maker because | love to > | e Because I've realized that nowadays, there are 
invent, come up with new ideas, and change the not many women engineers or women video- 
world. With so many great inventors today, in this game designers, so | would like to show the 
day and age, anything can happen and anything world that women are as smart, capable, or 
can be invented. It just depends on you. compassionate about technology as men are. 

e | feel that way because creating something from > | e | think after seeing the potential of what makers 
scratch is not my type of job/career | want to do can do I'm more interested in this subject. 
when | grow up. 

e Because | work with other people | don't know if | > | e | believed that as | am a part of FIRST robotics 
can call myself a maker just yet and that has given me the tools ability and 

opportunity to try being a maker and designer 

e | chose ‘Maybe’ because sometimes, in Art, | create | > | e | see myself as a maker because | made a cookie 
new designs, and in Science I've built a mousetrap cutter with the 3D printer, and now | am doing 
car, which ran for about 12.2 meters; so I'm not the social entrepreneurship project. 
sure if it follows in the designer/maker category. 


Students’ confidence in design-related activities. Sense of identity often goes hand- 
in-hand with a sense of confidence and self-efficacy. At the post and follow-up 
surveys, we asked students to share how confident they were overall in relation to 
several design-related activities. Students responded on a 5-point scale that included 
a neutral midpoint. Over half (54%) responded positively, and the average for the full 
sample was significantly higher than the neutral midpoint (mean= 3.6, sd=0.90; 
t(470)=13.831, p<0.001). Moreover, among the 43 students who completed both the 
post and follow-up surveys, there was a Statistically significant Increase in overall 
confidence, t(42) = 2.015, p=0.025. 


As with the question about identity, we also probed students’ confidence with an 
open-ended question. On the post survey, students who responded positively to the 
item about their confidence were invited to briefly elaborate on their choice. Two 
hundred forty-five students provided elaborations, and of these 186 reflected 
valuable reflections. Nearly a quarter (23%) of students cited an increase in skills as 
having contributed to their confidence. Examples included, “In the Learning Studio 
project | learned how to create 3D models on Tinkercad which allowed me to use 
the printer very effectively and the Sprout computer made it very easy to work with 
presentations’ and “| started to learn these skills by using the 3D printer and the 
Sprout. Before, | didn't Know what to do but, now i do.” A second trend was the 
connection between opportunities to practice, and increased confidence (20%). 
Students’ responses evidencing these two themes included, “| acquired lots of 
experience working in situations like these’; “I feel more confident with 3D modeling 


ou 


now that I've been practicing;” “Since we have worked on projects like this in here, | 
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am more comfortable using these skills;” “| got more comfortable with those things 
the more | used and applied them." 


A third and related theme was exposure to the Learning Studio equipment and 
experience itself (12%). “Because | got to test things and try different methods and 
listen to others’ ideas. | was exposed to more things than | think | would have been, 
and it has given me a chance to improve in these areas;" “| was able to explore with a 
different technology/equipment, that is more hands on, which gives you a different 
perspective to look at when solving a problem, or just doing everyday tasks.” and “I 
hadn't done projects like we've done in here before so it’s helped me learn how to do 
them and work with others on it.” 


Teamwork, learning from peers, and becoming more comfortable expressing 
themselves were also cited by students as reasons for why their confidence grew in 
the Learning Studio. *...before being in this class | didn't know how to really express 
my ideas and this class has made me try try to become a better inventor ...” “Since 
you work with other students you get a chance to open your social skills and when | 
teach things to other students it’s easier for me to understand and remember what | 
previously learned.” “...before | didn't like working with people. Now | work with kids 
that aren't my friends.” “Before | got to the Learning Studio, | would keep my ideas to 
myself...now that I've gone through the Learning Studio | share my ideas.” “We often 
work In groups to solve programming problems. Everyone usually has a different 
solution unless the problem Is dead simple, so | became comfortable abandoning my 
ideas in favor of others.” 


Several students also described shifts in attitude toward taking risks, being more open 
with others, and feeling more accepted by peers through their Learning Studio 
experience. Examples of students’ responses included, “| learned how to be more 
open with people and technology.” “Before | was worried about breaking the Sprout 
and 3D printer. Now that | know how to use it, I'm not scared of breaking them at all” 
"... before if | couldn't do something I'd give up especially with computers but now I'd 
at least try and fix it”. Interviews with teachers corroborated this theme. One high 
school teacher shared, “Tinkercad was new for me. When we were starting the 
prototyping and designing the cookie cutter, some of the girls were saying, ‘| hate 
computers, | can't do this’! | told them, I'm in the same boat. | didn't Know it either. 
And those same girls are now designing a special cookie cutter using Tinkercad for a 
special parent night at the school. It’s just amazing to see them progress even though 
they had these preconceived notions that they can't do something or aren't good at 
it. (...) Now my students are not as afraid to try something new, and if it doesn't work 
we'll go back and take a look... It's nice to see them break away from the stereotypes 
that they can't do something, and get out of that fixed mindset of ‘they've never been 
a computer person”. 
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3.4 Focal Outcomes 


3.4.1 Engagement & Persistence 


e Students are excited about coming to school 

e Students are eager to visit the Learning Studio 

e A diverse range of students demonstrates their enthusiasm for the Learning Studio 

e Students apply skills and perspectives from the Learning Studio to other aspects of life both in and out of 
school 

e Students seek deeper understanding and knowledge beyond the minimum requirements. 


Figure 17: Engagement and Persistence Indicators 


The first focal outcome for the Learning Studios was impacting student engagement 
and persistence. At the outset of the program, we identified several indicators to 
serve as operational definitions for these program impacts, as shown in Figure 17. We 
then gathered data from multiple sources to explore the extent to which the Learning 
Studio exhibited these indicators. 


Analyses from the surveys, as well as conversations with teachers and students 
provided evidence that engagement and persistence had been positively impacted 
through Learning Studio participation. From students’ responses to questions on the 
post survey, we learned that the majority of students endorsed frequently taking 
actions associated with engagement and persistence, such as continuing to work on 
an assignment beyond what was required (“keep working’), sticking with a tough 
problem until they solved it (“stuck tough"), and getting curious about how 
something worked (“get curious"). Figure 18 shows the distribution of students’ 
choices across the response options for each indicator. On four of the five indicators, 
approximately three in four (74%-80%) students responded in the frequent zone of 
the scale, from a few times to many times. On each of those four indicators, 
approximately half (from 48%-51%) the students responded with “several” or “many” 
times. 
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Keep Working (n=513) 


Solve Multiple (n=516) 


Stick Tough (n=518) 


Get Curious (n=520) 


WM Many times M#Severaltimes MA few times Acouple oftimes — Not really 


Figure 18: Frequency of Student Behaviors Related to Engagement and Persistence in the Learning Studio 


We found further evidence of impacts on students’ engagement and persistence in 
the open-ended responses they and their teachers provided on the post surveys. 
When asked to describe the ways they had grown in the Learning Studio, 6% of all 
students who completed the post survey cited aspects of persistence. Examples 
drawn from their responses include: 

e “That you can be creative if you tried really hard but you had to put all the 
effort you have;” 

e “Things might be hard sometimes you have to take a break from it but don't 
give up;” 

e “Patience, it can get frustrating trying to figure things out but once you stay 
with it, it gets easier. It is just hard in the beginning when you don't have the 
confidence or experience to get where you want to be." 

Teachers echoed similar observations when asked to share the specific ways that 
students had demonstrated engagement in the Learning Studio. Their post-survey 
responses reflected several of the indicators that had been independently set as 
operational definitions, including spending more time than was required in the 
Learning Studio, using time outside of class to work on their projects, and 
demonstrating unusually high levels of focus. Figure 19 presents a selection of their 
words. 
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e “Persisting through problems with their designs, finding humor while working through problems, managing impulses while 
collaborating, inquiring about other ways to accomplish tasks.” 

e “That sometimes learning is messy. Not all the time do our designs work on the first print. They also learned that it takes 
patience in order to persevere.” 

e “Students stayed focused and interested during the entire time while in the Learning Studio.” 

e “As a few of the students worked with the Sprout, they were reluctant to leave when the class ended. They also show focus 
on a level that | don't usually see.” 

e “They were excited to come to science and use the equipment. They also went home and talked to their parents about 
what they were doing. They were into the class from the moment it started.” 

e “They attend classes regularly on a voluntary basis, bringing with them more and more friends Spend hours at home 
preparing material for the next lesson (e.g. drawing backgrounds and characters for stop-and-motion)" 

e “Students choosing to spend their free time in the Learning Studio was huge and showed true, authentic engagement.” 

e “High school students vote with their feet...if the release bell rings and nobody moves (instead my space starts filling up) 
you have authentic engagement in learning.” 

e “Many of the students involved in the Learning Studio continued their work in the classroom and at lunchtimes in the 
Learning Studio. There was also a large uptake of students using Tinkercad at home.” 


Figure 19: Teachers’ Examples of Student Engagement and Persistence in the Learning Studio 


In interviews as well, teachers noted several ways they had observed students’ 
engagement and persistence develop in the Learning Studio, and shared examples of 
those mindsets translating to other classes and subjects. “I’ve seen that maker 
mindset grow over the year and it's really powerful because it translates into math, 
and it translates into reading. It’s not just limited to the making that we're doing. 
When they come across a problem that's hard in math they understand that, ok, it is 
hard, but it’s a challenge that I’m not good at yet. | can improve, | have to work, | have 
to practice...they embrace challenges a lot better than they had if they hadn't had 

[the Learning Studio]. The whole saying-something-is-too-hard-and-giving-up 
doesn't really apply to them too much.” 


A high school teacher described a benefit of the Learning Studio as helping, “to 
remove the fear of the creative. Because schools have this tendency to take away 
imagination. Everything is black and white. They worry about perfection, but that's 
not how the real-world works. So [student] lose their creativity, they lose their 
problem-solving skills. They lose the imagination and are afraid of entering the ‘what 
ifs’ scenarios, the ‘how might we’ questions, because they focus more on themselves 
than on others. So it helped to develop collaboration and communication skills as 
they are teaching each other how to do things. It helped them learn to take some 
risks, particularly when it’s computer based and it doesn't work out. You erase it, you 
delete it. That's not a big deal. The only question | would ever ask is, ‘what did you 
learn from it’? So they learn to get comfortable being uncomfortable, and they learn 
to analyze their failures, not just quit but to persevere through them, and through 
iteration learn to improve.” 


A number of teachers commented that student growth in the Learning Studio had 
ripple effects into other classes and domains. “Last year | did Maker Mondays, so 
every Monday there was an hour dedicated to Making. This year | just kind of 
embedded it throughout. This is my second year, and my first year with the Learning 
Studio and that cool technology. [The students] go to science once a week and the 
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science teacher has been doing a lot of STEM activities and challenges and she said 
that she can totally see a difference in the kids in my class this year, and even the kids 
| had last year, when they encounter some of those STEM challenges or any kind of 
making or problem solving, that they handle it a lot better, they persevere, they don't 
give up, they come up with some really creative ideas and they feel confident about 
them. You can see a difference and that really inspires me to keep going.” 


Finally, on the follow-up survey, we asked teachers to indicate on a 5-point response 
scale, from strongly disagree to strongly agree, whether they felt students had put in 
more time than was required, and had found personal meaning in their Learning 
Studio projects. The response scale included a neutral mid-point. Among the 27 
teachers who completed the follow-up survey, the great majority (85%) agreed that 
students had invested more time than required, and nine in ten teachers (89%) agreed 
that the projects had personal meaning for students, with 44% strongly agreeing. In 
addition, the average responses on these two items were also significantly above the 
neutral mid-point of the scale, t(26)=6.596, p<.001; t(26)=10.198, p<.001, 
respectively. 


3.4.2 Agency & Ownership of Learning 


e setting personal learning goals 

e seeking and responding to feedback 
e taking intellectual risks 

@ personalizing projects or assignments 


Figure 20: Agency and Ownership of Learning Indicators 


The second focal outcome we anticipated observing in students was an increase in 
their agency and ownership of learning in the Learning Studio. As with engagement 
and persistence, at the outset of the program we crafted a set of indicators, listed in 
Figure 20, that could serve as operational definitions for this outcome. From analyses 
of the surveys and interviews, overall we found support that the Learning Studios 
promoted this aspect of a maker mindset. 


Affording students opportunities to exhibit personal agency and ownership of their 
learning is not common in traditional classrooms for a variety of reasons. One 
aspiration of the Learning Studios program was to contribute to a culture of learning 
in which students take action to drive their own growth. This translated to anticipated 
changes for teachers as well as for students. At the follow-up survey, teachers 
expressed nearly unanimous (96%) agreement that through their participation in the 
Learning Studio, they are now more comfortable leading projects in which students 
have a high degree of agency. 
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To capture the point of view of students on these outcomes, we included five 
specific actions related to agency on the post survey and asked for ratings of how 
often they engaged in them during their time in the Learning Studio. As shown in 
Figure 21, across all five indicators, approximately two-thirds of students reported 
carrying out the target behaviors from a few times to many times, with over 40% 
reporting these behaviors several to many times. Moreover, the vast majority (84%- 
86%) enacted them to some extent. 


Take Risks (n=520) 


Set Own Goals (n=520) 


Choose to Lear (n=522) 


Give Feedback (n=521) 


Seek Feedback (n=523) 17% 


0% 20% 40% 60% 80% 


MMany times WSeveraltimes MA few times A couple of times ' Not really 


Figure 21: Frequency of Student Behaviors Related to Agency and Initiative in the Learning Studio 


To further probe the extent to which students gained facility in seeking, providing and 
responding to feedback, we posed additional questions to that effect on the teacher 
follow-up survey. The great majority of teachers, responding on a 5-point scale that 
included a neutral mid-point, felt that their students had grown in their active use of 
feedback on their work. These data are presented in Figure 22. 
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Through participation in the Learning Studio, students gained facility in... 


Requesting Feedback Integrating Feedback Offering Feedback 


O. Oe OX 


Figure 22. Teacher Ratings of Growth in Students’ Feedback-Related Indicators of Agency 


Evidence of growth in students’ initiative in the Learning Studio also surfaced 
frequently in interviews with teachers over the course of the program. One high- 
school teacher shared, “Earlier on, they would come in looking for the one correct 
answer. ‘Is this right’? And I'd tell them, ‘| don't Know -- You're asking me if it's right, 
you haven't even told me what the question is. How did you come to this answer? 
What evidence do you have to support this answer? Your answer may be right, try it’.” 
The students initially are ‘horrified’. “Then they start realizing that | am teaching the 
way It works in real life, and that if you work for me, you come to me with solutions, 
now you're valuable. That's something that they struggle with.” By the end of the 
year, “they don't really have problems with it anymore. They get used to it and it helps 
them in their core classes also. (...) The design-based system gets them to take 
responsibility and accountability and ownership of what they're working on.” 


A middle school teacher explained, “It's really student guided, led and directed. We're 
finding out that the engagement is increasing and they are starting to understand that 
if they don’t know how to do something they go to the Sprout and see if they can 
figure it out there. (...) Students are also managing their risk as they work. They say, 
well if we do this are we going to lose time? There's an increase in planning and 
managing impulsivity. Most of that learning came from the Parts, Purposes and 
Complexities project [that Digital Promise Global provided, based on the thinking 
routine created by Agency by Design]. They got to choose what they wanted to take 
apart. Rather than ripping things apart, they first sat down, thought about how it was 
put together, took pictures, identified which tools they would need first. Whereas in 
the past they'd have a hammer out right away just to deconstruct something. (...) 
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There's also been an increase in inquiry - students wondering about things, asking a 
question then going in a direction to find their answers." 


This theme was echoed by others. For instance, a high school teacher found that 
with students’ increased comfort came increased independence. “Early on in this 
process, more often than not | was focused on the tech and the process, guiding 
them in the correct steps through their process and reassuring them that it was okay 
to fail. And my role has shifted recently to more of someone they can bounce ideas 
off of, and talk more about their ideas and more about process.” And another 
highlighted the importance of providing students with options to set their own 
learning goals. “| continue to praise the open-endedness of this process and of the 
Learning Studios. | have students that are learning about topics that are never going 
to be covered in a standard but they are important to them and they are learning 
about problems that are going on around the planet and country....they are able to 
stand up and speak very intelligently about their topic. It’s empowering to them and 
certainly motivating for me.” 


The principal of an elementary school explained, “The main area of growth that | 
witnessed was in the children’s ability to think outside the box, once they got the idea 
that they were allowed to think outside the box. The children are so used to being 
rooted in fairly traditional pedagogy. So when you suddenly say, ‘we're going to do 
Learning Studio and you're going to work in groups and come up with solutions to 
these challenges... That's a big, big change from the teacher standing in front of the 
group directing the activities.” 


3.4.3 Empathy, Collaboration & Communication 


e Awareness of the importance of taking perspective and understanding where others are coming from 

e = Ability to communicate effectively in a group work environment 

e Ability to work effectively in a team, such as recognizing strengths that others bring, compromising and 
working through disagreements 


Figure 23: Indicators of Empathy, Collaboration & Communication 


Design thinking and making do not take place in a vacuum. The ability to 
communicate one's ideas, recognize others’ strengths, navigate disagreements, and 
understand where others are coming from are all critical competencies. The third 
focal outcome for the Learning Studio targeted these abilities for communication, 
collaboration and empathy, defined in operational terms using the behaviors in Figure 
23. Overall, we found support for Learning Studio contributions to students’ growth 
in their collaboration and communication abilities among a subset of students. 
Whereas ceiling effects on survey items limited insights about impacts in this area, 
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open-ended responses from students and teachers on surveys and in conversations 
generated many anecdotes of growth, especially for teamwork and collaboration. 


Both teachers and students responded to survey items regarding various aspects of 
collaboration and communication. Specifically, we asked them to rate their personal 
comfort levels, on a 5-point scale from “Not at all comfortable” to “Totally 
Comfortable” for: communicating new ideas to others; being open to new and 
diverse perspectives; incorporating group input and feedback into work; and letting 
go of your idea in favor of an idea more supported by members of your team. In 
addition to rating their personal comfort, teachers were also asked to Indicate their 
comfort facilitating students in these areas. 


From students’ pre to post surveys, we found marginally significant gains on being 
open to new and diverse perspectives; however, responses for the other 
communication and collaboration indicators remained fairly constant over time. 
Among the small group of students who completed all three surveys, results were 
mixed as well, with significant changes found only for “letting go of your idea in favor 
of an idea more supported by members of your team”. Appendix D presents these 
results in detail. 


Since these response trends did not reflect findings from interviews and open-ended 
responses, we took a closer look at the distributions across the full set of responses 
at each timepoint. This revealed what is known as a “ceiling effect” in the student 
data. As can be seen in Figure 24, at the outset of the project, fewer than one in five 
students reported low comfort levels for the indicators. In other words, with 
approximately 80% of students already expressing their comfort on in these areas, 
there was little room for improvement. Notwithstanding, for all four indicators, the 
percentages of students in the “low comfort" zone were lowest at the follow-up 
survey. 


To get a better understanding of the extent of growth for these outcomes, we filtered 
for students who reported attending the Learning Studio at least 10 times. For these 
four items, we calculated the number of students who moved to higher comfort 
levels from pre to post. For students whose pre-survey responses were ‘3’ or lower, 
37% showed growth for communicating new ideas to others; 36% for incorporating 
input; 51% for being open to new perspectives; and 30% for letting go of one’s own 
idea in favor of someone else's. 


’ Digital Promise Global Learning Studios Research 


40 


Communicating new ideas 


0% 20% 40% 60% 80% 100% 


pre SN 20% % 
POST SN 13% 7% 
OL a 14% 4% 


M4(High) M3 2 W1(Low) 


Being open to new and diverse 
perspectives 


0% 20% 40% 60% 80% 100% 


pre SZ 20% 9% 
POST SN 18% 7% 
OL TN 14% 4% 


Incorporate input 
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Letting go of one’s own idea in favor of 
another 


0% 20% 40% 60% 80% 100% 


pre SZ 20% % 
PosT SA 13% 7% 
OT 14% 4% 


M4(High) M3 W2 MW1(Low) M4(High) M3 2 MH1(Low) 


*Data shown represent the full set of responses at each survey 


Figure 24: Distributions of Student Responses for Communication and Collaboration Indicators+ 


Teachers’ responses to the same questions followed a different pattern, but also 
evidenced a ceiling effect. From pre to post, teachers reported personal growth on 
two of the four indicators: communicating new ideas to others, t(19)=2.517, p=0.01, and 
incorporating group input and feedback into work t(19)= 1.831, p=0.041. On average, 
teachers’ comfort communicating new ideas to others was also higher at the follow- 
up than at the pre-survey. Examination of the distribution of responses to the 
questions showed that from the outset, teachers were highly confident in their ability 
to facilitate students in these areas. Of the 27 teachers who responded, only one 
expressed low comfort at any of the time points for each indicator. Moreover, at the 
pre-survey, between 60% and 85% indicated high levels of comfort. 


Because of the observed ceiling effects on these items for students and teachers, we 
focused our interpretation on related questions from the post and follow-up surveys, 
as well as on open-ended survey responses and interview data. These analyses 
yielded substantial evidence that the Learning Studio had positively contributed to 
students’ growth in communication and collaboration. On the follow-up survey, we 
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asked teachers to share their perspective on whether students had improved their 
collaboration and communication abilities through their participation in the Learning 
Studio. Of the 27 teachers who responded, virtually all agreed that their students 
became better collaborators, with nearly two-thirds (63%) strongly agreeing. All but 
one teacher agreed that their students had improved in their ability to communicate 
their ideas to others, with over half (55%) strongly agreeing with the statement. 


A second source of quantitative data on these outcomes were students’ ratings of six 
indicators used to measure students’ community and social support behaviors during 
their time in the Learning Studio. Students reported frequently incorporating ideas 
from their peers into their own work, helping peers to brainstorm, providing support 
to a peer, and encouraging a peer, as shown in Figure 25. Well over half also reported 
working with someone new. 


Learn from Classmate (n=514) 


Encourage Peer (n=519) OZ icx liz 
Use Ideas From Peer (n=521) Wc 3 isx OR) 
Work with New Peer (=519 I TScsecNn 19%  —NNNDISCNNNN 


Help Brainstorm (n=520) Zs 21% = 
Help Peer (n=521) a 198% 11% 


0% 20% 40% 60% 80% 


MMany times MSeveraltimes MA few times A couple of times &@ Not really 


Figure 25: Frequency of Student Behaviors Related to Community and Social Support in the Learning Studio 
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e | learned how to work with people that | may not like or may not like me and how to be more accepting of 
other people/teammates ideas or observations. 

| learned that teamwork is an important part of using the Learning Studio. 

| learned how to work with others and be more open minded to different ideas. 

That working on a project together instead of separate is more effective. 

You need to respect each of your team members’ ideas. 

| learned that being in a group can be hard to work with because sometimes YOU want to take charge on 
the project and group but sometimes you have to listen to others and see what their ideas are, and that 
there is a loti can do with a 3D printer to change the world. 

| learned to always keep an open mind, to not shoot down ideas that haven't at least been tested. 
Cooperation is almost always a key to being successful. 


Figure 26: Examples of Collaboration from Students’ Open-ended Survey Responses 


We found further evidence of students’ improved collaboration skills in their open- 
ended survey questions. Fourteen percent of students, when asked what they had 
learned in the Learning Studio, cited collaboration and teamwork as an area of 
growth. Examples of their responses appear in Figure 26 and reflect perspective- 
taking, openness to new ideas, and the value of working collaboratively. 


Interviews with teachers also yielded many examples of student growth related to 
communication and collaboration. A high-school teacher commented, “They have 
gotten better and better at listening to one another. Just being in a position where 
they are being asked to collaborate constantly, where they are being assessed by 
their peers on that collaboration -- something I've put in place this year. They have to 
really lean on each other.” The principal of a traditional elementary school reflected, 
“We found that one of the road blocks early on was their lack of collaborative skills. 
What we've seen over the course of this last term is the improvement there. 
Generally, when we started, we were doing basic STEM challenges and some kids 
had literally no idea how to work as a group. Teachers have noticed that there has 
been significant growth in a number of children who started out quite anti-group 
work and have realized that you have to work with a group to be successful." 


Regarding communication skills, another elementary school teacher shared, “The 
group work is so much stronger because each of the kids Is advocating for their 
thought process... They have to think about things from different angles, and they 
have to think about what do other people value and so their communication skills are 
building because they can't just come from their perspective.” The teacher also 
described how students have learned to value the contributions of their peers. “| 
brought 20 kids to present in [city] this year and in preparing the kids for questions, | 
was able to hear from them what they felt was important. One of the things that | 
liked the most was the kids talking about how much they learn from each other." 


One high-school teacher described the difference between group-work in typical 
classrooms, and collaboration in the Learning Studio. “A lot of teachers will assign 
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what they call ‘group work’ and group work doesn't work. One person does 
everything, or one person is insistent that it has to be done their way. | don't do that. | 
tell them, you have to figure out what each other's strengths are and utilize those 
strengths. So one person is going to design, one person is going to research, one 
person is going to build. But while you're doing that, if you're primarily responsible for 
the building, you still have to know the research and the design...and you see them 
come together...and all of the sudden, you'll find the builder becomes the better 
designer, the designer becomes the better researcher, the researcher is becoming 
the better builder.” 


Teachers also described students reaching out to each other for help and input. “The 
kids are talking to other kids who have used [the technology]. Other kids are saying 
‘yeah, we did this but that didn’t work out so well, I'd recommend doing it like this...” | 
find myself asking the kids, too. ‘Has anyone ever done this before?’ and usually 
there's someone in the group who will say, ‘Yeah we did that but it didn’t work 
because of this’...and that's the more important part: being able to articulate why you 
think something is, not just, | don't like it or | like it.” 


3.4.4 Design Thinking & Problem Solving 


e Recognizing that the material world is designed; 

e Ability to identify problems to solve; 

e Ability to take perspective on one’s own creation and those of others; 

e Demonstrating variation of efforts; and, 

e Ability to recognize failure and iteration as a regular part of the design process. 


Figure 27: Indicators of Design Thinking and Problem Solving 


The fourth and final focal outcome we anticipated for participants in the Learning 
Studio was growth in design thinking and problem-solving abilities. Key markers, 
which served as operational definitions, appear in Figure 27. Overall, we found some 
positive impacts on students’ beliefs and attitudes in this area; however, evidence of 
growth in students’ awareness of and facility in design-based processes was mixed. 
On the one hand, quantitative responses from students and teachers regarding 
design-related activities showed increases, and interviews with teachers offered 
some examples of students engaging in design processes. At the same time, few 
differences were found from pre to post when students were asked to describe a 
designer or maker in their own words. Moreover, the great majority of students’ 
solutions to three distinct design-thinking prompts did not demonstrate a grasp of 
key characteristics of the design process, such as developing an understanding of the 
problem space, developing prototypes, testing and iterating the design. In this 
section, we lead with the quantitative data, and transition to the qualitative sources of 
data from surveys and interviews to explore impacts on design thinking and problem 
solving. 
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Quantitative Findings. As with the other focal outcomes, we posed questions to 
teachers and students about their comfort levels with processes related to design 
thinking and problem solving. Teachers were also invited to indicate their comfort 
facilitating their students in these design processes. Results for students’ responses 
are presented in Table 12. From pre to post, main effects were significant for one of 
these indicators, though the average at post was still below the threshold for 
“comfortable”. Moreover, students’ reported comfort taking something apart dropped 
significantly from pre to post survey. When these questions were posed again in the 
follow-up survey, trends for most items were generally positive from pre to follow-up 
but tended to be relatively lower at the post survey. The repeated measures analysis 
detected at least marginally significant differences for three items across all three 
time points. 


Table 12: Students’ Comfort with Design Thinking and Problem-Solving Activities++ 


PRE POST FOE 
Mean sd Mea sd Mean sd PRE-POST PRE-POST-FOL 
n 

Defining problems to 2.81 795 2.88 .814 t(304)=1.310 (ns) 
investigate 

S105 707 3.18 .683 sro .694 F(2,76)=2.634° 
Taking something apart to | 3.22 .847 3.06 .869 t(304)=-2.936 
see how it works 

3.03 915 2.95 .837 Sls .906 F(2,74)=.590 (ns) 
Assembling something 3.47 75S) 3.48 144 t(303)=.205 (ns) 
WITH instructions 

S155) .602 3.45 .602 SA soils) F(2,74)=2.552° 
Assembling something 249 | 992 | 2.63 | .986 t(303)=2.791" 
WITHOUT instructions 

2.26 891 2.61 916 2.95 | .899 F(2,74)=12.448°— 
Working effectively when 2.74 821 2H) 865 t(304)=.686 (ns) 
process is ambiguous 

3.05 .695 3.03 5S 3.16 .916 F(2,74)=.471 (ns) 


** Response scale: 1 (Low) - 4 (High) 
“9<.001; p<.01; *p<.10 


For teachers’ personal comfort with these design-related processes, a clear ceiling 
effect was detected. Nine in ten teachers reported moderate to high comfort on the 
second and fifth indicators; 73% to 79% reported positive comfort levels on the other 
three indicators. Because of the high response levels at the pre-survey, there was 
little room for additional growth during the Learning Studio implementation. 
However, some increases were found to be statistically significant from pre to post. 
For instance, teachers reported increased comfort in defining problems to 
investigate, t(19)=2.549, p=0.015 and assembling something with instructions, 
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t(19)=2.333, p=0.015. Regarding teachers’ comfort facilitating students in these 
activities, the only significant increase from pre to post was in assembling something 
using instructions, t(14)=3.228, p=0.003. 


On the follow-up survey, we asked teachers two additional questions about this focal 


outcome. All but one teacher agreed that they were now more skilled at leading 
students in the design process, with 37% strongly agreeing; 93% of teachers agreed 
that students’ perspective-taking ability had improved, with 44% strongly agreeing. 


Qualitative Findings: Definitions. The pre and post surveys included several open- 
ended prompts to students to respond in their own words. One question asked 
students to describe a designer or maker in their own words. Each students’ pre and 
post responses were compared for differences. Of the 401 students who provided a 
response at both time points, 61 were scored as having improved definitions in some 
way. Select examples are shown in Table 13. Among the improvements found from 
pre to post were references to problem solving, persistence, a maker mindset, and 
improving the world. Several students’ responses suggested that they had not 
become familiar with the term designer or maker, and interpreted the term in the 


context of fashion or art. 


Table 13: Examples of Students’ Pre-Post Definitions of a Designer or Maker 


PRE POST Improved? 
Any person who is thriving and creative. A designer is someone who solves a problem. Yes 
No matter how big or small. They are 
innovative and prepared. 
Someone who can take simple everyday things | A maker is someone who uses technology to Yes 
and create something new and abstract out of create new things. They may improve on 
them. For example, wood is just wood unless existing inventions or create solutions to new 
you make it into a chair or a house. problems. 
A person that have no limits from what they are | | see a designer as an very creative person to Yes 
making and creative freedom. come up with design that help people in their 
everyday lives. 
Creative. A designer/maker is someone who is able to Yes 
design and make something. They usually need 
some creativity and are able to reach goals. 
| would describe them as very smart person Someone who is intelligent and someone who No 
who knows what they're doing with is smart about their materiel making. 
tools/computer, basically anything. 
To me a designer is someone who comes up A designer is a person who designs a product No 
with a design for a product and a maker makes and a maker is a person who makes the 
the actual product. product. 
| will describe them as inventor and someone Someone very creative and one of the smartest No 


with a huge imagination. 


groups of people 


Y Digital Promise Global 


Learning Studios Research 


46 


Qualitative Findings: Design Scenarios. To determine whether students had 
progressed on design thinking processes beyond self-report or their teachers’ 
perceptions, the pre and post surveys for students included three design scenarios. In 
the first item, students were shown a fan and asked to describe how they would 
figure out how it works, using as much detail as possible. The second question 
showed a three-dimensional image of a 3D-modeled car and asked students how 
they would model the object in a program like Tinkercad. The third question 
described an everyday problem scenario: “When students at the Imaginary School go 
outside for recess, they have an area on the floor of the cafeteria where they leave 
their bags. As bags get thrown and piled on top of each other, students are finding 
that things inside them are getting broken.” For this third scenario, students were 
asked to explain, in as much detail as possible, how they would go about developing 
a solution to this problem. 


Analyses of students’ responses to these questions suggest that the majority of 
students did not develop explicit knowledge of a method for design thinking. For the 
fan item, roughly one in five (22%) of students showed at least some improvement 
from pre to post, with just 2% showing strong improvement; 78% did not show 
evidence of growth. Examples of pre-post responses scored improvement as strong, 
moderate, or none are shown in Table 14. 
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Table 14: Examples of Students’ Pre-Post Solutions to the Fan Scenario 


PRE POST Improved? 
The propellers show that they The first thing that | would do is try and see what the Strong 
move quickly because they are in purpose of the machine is, then | would try and take apart 
a cage so | assume that it will give | the cage and see if the blades spin, and if they don't then | 
of wind. would take the battery out and try to see how that works. 
The blades rotate which creates a | would first examine the outside of the device, then | would Strong 
breeze. take it apart and look at the insides and try to find out how 

they are connected and work together. 
It works as a function, you | would take it apart, and try to put it back together. | would Strong 
press/put something into it. And examine all the parts and see how they work together 
then you get the results, in this before actually trying to rebuild it. 
case the fan blows out air. 
Ask siri google to see how it You press a button and it starts the engine and it makes the Moderate 
works. fan work. 
A switch would trigger the fan to Pull off the cover and then you pull the fan off and then you Moderate 
start going by blowing air to cool. search the inside bit. Then put it together again. 
You can find a button, search on | would look inside and | can see a fan so it gives wind to Moderate 
google or inspecting it in detail. you when it’s hot. Its covered with metal bits around the fan 
and that is to not hurt you. 

| would take the object apart and | would take apart the fan and look at all the parts. None 
look at the parts then try to piece 
them together to find out how the 
thing works. 
| would find the button. | would find the plug in and turn on the on button. None 


The second scenario asked students how they would model the 3D car shown in an 
image. Overall, stronger outcomes were found for this question, which focused on 


technical aspects of designing and making. More than three in ten students 


demonstrated improved responses from pre to post, with 5% showing especially 
strong growth; for this performance task, 65% of students’ responses did not 
demonstrate growth. Table 15 provides selections of responses. 


Y Digital Promise Global 


Learning Studios Research 


48 


Table 15: Examples of Students’ Pre-Post Solutions to the 3D Modeling Scenario 


PRE POST Improved? 
How | would model the object | would model the picture in Tinkercad by getting two Strong 
above is to shape the car with cylinders, a trapezoid, and a rectangle that has two holes 
different pieces to form the car due to the cylinders going through. Place the two cylinders 
shape. near the edges of the rectangle so holes are created and put 
the trapezoid on top. 
| do not Know what Tinkercad is, | would start off with a rectangle and then use holes to Strong 
but | hope | will get to learn what shave off pieces of the rectangle and give it the shape of a 
it is and how to use it. car, | would use about two holes to give the car the shape 
on the roof of the car as depicted in the illustration, and 
another four circular holes to make space for four solid 
circles to replace the holes and act as wheels. 
Use a trapezoid, a rectangle and 2 | Grab two cylinders and place them horizontally, grab two Strong 
cylinders. arches and stretch them to the length of the cylinders, then | 
would grab a trapezoid and place it on top of a rectangle on 
the bottom in between of the arches. 
| don't know yet. Create that figure using the different objects the program Moderate 
gave us and all the options they let us use like grouping. 
| would learn more about it. you bring the object to the work space and put them to the Moderate 
size that's asked. 
it is a car. | would use shapes and group them. Moderate 
make a car out of cardboard and Build it out of clay put it in the 3D printer and wait for it to None 
other stuff then scan it with the print. 
sprout. 
First | would find wheels for a | would first have to create a model then make it piece by None 
substitution and use cardboard to | piece. | would first make the body and then the wheels and a 
create the body. Then | would stick like object to connect the wheels. 
colour or paint everything else. 


The third scenario focused on problem-solving in a real-world situation that would 


have required developing an understanding of the context, interviewing key 


individuals such as students and administrators, and then developing, testing, and 
iterating a solution. Roughly 9% of students’ responses evidenced some improvement 
from pre to post, with a handful of strong examples, as illustrated in Table 16. 
However, the majority (91%) showed no improvement. 
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Table 16: Examples of Students’ Pre-Post Solutions to the 3D Modeling Scenario 


PRE POST Improved? 
| would tell the students to put | would determine what the problem is, which is that items Strong 
their bags in the locker or put are getting broken. | would then find out what has caused 
their valuables in their lockers. the problem. The items are breaking because students are 

piles bags on bags. | would then find a solution that would 
prevent things from breaking. The students can put their 
bags in their locker. They could also put their bags along the 
wall so that there isn't a pile. 
Have separate areas for Find out what is causing the problem, then brainstorming Strong 
individual bags. solutions to the problem, eventually picking one that suits 
best and is the most efficient. 
Place them neatly on the side. | would use Tinkercad to create hooks for each student and Strong 
use the 3D printer to obtain them so they kids can hang up 
their bags. 
Try to make something out of it | First | would look at the destroyed objects inside the bags. Moderate 
like a giant robot. Next, | would think on how to fix it. Finally, | put the idea to 
work. 
I'd make cubby with | would make a cubby design on Tinkercad then print it out Moderate 
everybody's names on them as a prototype. Then | would gather wood and build cubbies 
that are big enough to fit their and engrave each of their names on their cubby then they 
bags and when the go out they would each have a cubby. Or if buy lockers. 
are in number order so the each 
put their bags in their cubbies 
and go out. 
| don't KNOW...maybe Testing materials that would hypothetically keep stuff from Moderate 
modifying the bags so they can breaking and modeling redesigns for either the floor or 
protect the content they carry backpacks that would prevent future breakage. 
more effectively??? 
You might solve this problem by | A solution might be that there are cubbies made and each None 
creating assigned cubbies so student is given a number and that student has to put all of 
things are not broken when their things unneeded for recess in that cubby. 
slapped against each other in 
book bags. 
Use shelves. Build a shelf. None 


Qualitative Findings: Teacher Interviews. Conversations with teachers about students’ 
design thinking offered additional insights. From these teachers, we heard examples 


of several indicators for this outcome. For instance, some teachers described an 

increase in recognizing that the material world is designed. One teacher reflected, 
“You just start to see that they see the world a little differently and now when they 
have an idea, they're like, ‘| can make that’ and that’s kind of cool.” 


We also found that in classes where a design process was already present—for 


instance, for students following a Project Lead the Way curriculum, the technology 
enabled them to focus more on the design process and iterations. “The equipment 
itself, like the Sprout, has prompted a different skill set, just working in that virtual 3rd 
dimension. Or being able to look at a project from different sides. Manipulating those 
programs for the 3D printing has given them access to new skills, or to try new 
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things”. An elementary teacher explained how the touch screen enabled her 
youngest students to interact effectively with the design software—something that 
was not possible when reliant on a mouse. From a design point of view, access to the 
Sprout and 3D printer meant students could focus their attention on the design 
process, iterating more quickly than if they were working exclusively with physical 
materials. Teachers described the technology as an asset for creativity and 
innovation. “Giving them new tools is giving them the realization that they have the 
Capacity to go out and learn. They never would have dreamed they'd be 3D printing 
or be able to design something that could be printed on a 3D printer." 


One high school teacher attributed an increase in the complexity of student work to 
access to the technology. “Some students are working on NASA project, where they 
are asked to design the spacecraft that is going to take a man to Mars. They've been 
designing it in Tinkercad and I've been working with them to get that printed. What 
they've been able to accomplish and their attention to detail is rather staggering. ” 


3.5 Students who Struggle 


In schools around the world, many students struggle in a variety of ways, and fora 
variety of reasons, to succeed academically. For some, the format of traditional 
classrooms fails to capture their interest and energy. Others face learning disabilities 
related to cognitive processing or attention. Because the Learning Studio program 
aimed to foster an interactive, collaborative learning environment that differs from 
typical classrooms, we anticipated that students who struggle at school might find 
new opportunities to engage and experience success. We explored this expectation 
through closed- and open-ended survey questions as well as in interviews with 
teachers. By teachers’ estimates, well over 300 students with documented learning 
disabilities were served by the Learning Studios program. Overall, we found good 
support that the Learning Studios provided opportunities for students who struggle to 
discover new interests, to develop valued expertise, to grow in their confidence, and 
to build positive rapport with their classmates. 


Learning Studio Benefits. A first indicator of this area of impact was teachers’ strong 
endorsement on the follow-up survey with the statement that some students who 
are not typically engaged in classes really got into the Learning Studio projects. Nine 
in ten teachers agreed, with 41% strongly agreeing. The post survey asked teachers to 
elaborate on the Learning Studio experience of students with diagnosed learning 
disabilities (L.DD), and interview questions also probed teachers’ observations in this 
area. 


A strong theme that emerged from teachers’ written responses, as well as in 
conversations, was the opportunity for LDD students to develop expertise that gave 
them confidence as well as new status among their peers. As one teacher wrote, 
“Some of the students with learning difficulties found ways to express themselves and 
assist their groups in non-traditional ways.” In interviews, another teacher described a 
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student who struggled with reading and low self-esteem. “She was so proud that she 
got hers done and understood it. You could just see her eyes light up and her 
confidence build. And then she could share and help others.” This observation was 
echoed by another elementary school teacher, who remarked, “The kids are not all 
good at the same things, but If you find those kids who are 3D experts, now they 
have something they feel they can contribute to the whole. It’s adding those options 
and areas for kids to find their expertise.” 


At one elementary school site, teachers and students were especially struck by the 
unexpected emergence of an autistic student as the school expert. “It was new for 
him to be such a help -- for him and for everyone else. That opened up a lot of eyes 
in the class. The student told me it was the best thing he’s ever done in school. 
‘Because I'm helping everybody. | feel really good because I’m good at it.’ This theme, 
of students seeing themselves, and being seen, in a different light through the 
Learning Studio experience was found across grade levels. A high-school teacher 
explained, “You may not do well in a traditional classroom, where you feel like there's 
a right or wrong answer and it becomes easy to be discouraged if you always get the 
wrong answer. Take [student], for example. She tries really hard, but she’s also very 
susceptible to saying, ‘| don't get it, | don’t belong here’. For the hour of code, | asked 
them to come up with a way to do code a button that looked like a lion and played a 
lion sound. There are a lot of ways you can do that, so it provides an opportunity. She 
came up with a different way of doing it that was right. That moment where she gets 
to see that she came up with something that the teacher didn't come up with.” 


We also found some evidence that the confidence LDD students gained in the 
Learning Studio flowed over into their other classes. For example, a high school 
teacher recounted, “They're used to being told ‘you don't get it’ repeatedly. And | 
would just ask questions, ‘you know how to use the apps on your phone? How'd you 
learn those? Find what you like that you're curious about and explore it.’ Well, they 
found that they enjoyed it. That really boosted their confidence and their problem- 
solving skills, their communication skills, and as a result, the reports | get back from 
their other classes Is that they started doing better and framing things differently 
there." 


Challenges to Engaging Students Who Struggle. From open-ended survey responses 
and interviews with the focal site teachers, we concluded that for the most part, 
teachers did not experience insurmountable challenges specifically related to 
working with LDD students in the Learning Studio. Of the twenty teachers who 
responded to this question on the post survey, half indicated that no unique 
challenges had been encountered. Among the ten who did describe challenges, the 
responses were varied. Common themes included the need for extra time and 
additional support. “Initially, these students found it difficult to add value to their 
group as social and emotional barriers prevented them from really being part of the 
group. However, as their ICT skills developed, they became Known as the group's go 
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to person for help with Tinkercad or Powerpoint which tn turn provided them with 
more respect from their peers and a higher level of confidence.” Another teacher 
wrote, “They just needed a little additional time and support. Many times they were 
worried about doing it ‘wrong’ but once they figured out they could do it whatever 
way was best for them, they excelled.” 


3.6 Student Outcomes Associated with Implementation Characteristics 


As described in Section 1.2, what the Learning Studio program had in common across 
all sites was the provision of an advanced technology package for designing and 
making, access to an online professional learning community, and a set of project 
guides and Challenge projects which teachers could choose to implement at their 
discretion. This resulted in marked variation across sites in terms of which projects 
were carried out, the ways in which Learning Studio activities were integrated into 
students’ broader curriculum, and the nature of the guidance and instruction 
students experienced. Another factor distinguishing each site was the unique 
background each teacher brought to their respective Learning Studio 
implementation. To explore whether these variations in Learning Studio 
implementation sites were associated with differences in student outcomes, we 
carried out two sets of analyses. The first focused on project implementations; the 
second examined the effects associated with teacher background and experience. 


Teachers’ Use of Project Guides and Challenges. We first looked at student outcomes 
in relation to the number of projects their teachers used in the Learning Studio during 
the first phase of implementation, through December 2016. In this context, “projects” 
refers specifically to the project guides that the Learning Studio program staff 
developed and offered to teachers. Examples of the project guides appear in 
Appendix A. In addition to the culminating Challenge project, a total of ten project 
guides were available to teachers. Four of these were suggested for “project of the 
week" in a sequence. To calculate the number of projects completed, we linked 
teachers’ responses about implementation to student data. We then examined 
correlations between the number of projects each student completed with their focal 
outcomes. To provide context for this analysis, Figure 28 presents the distribution of 
teachers’ responses to the number of Learning Studio projects they completed with 
their students. 
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Figure 28: Number of Learning Studio Projects Implemented, by Teacher 
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Results suggest that for three of the four focal areas, the more Digital Promise 
provided project guides a teacher implemented with students, the stronger their 
students’ outcomes were (See Table 18). For engagement and persistence indicators, 
every indicator had positive and significant correlations, ranging from r=0.103 to 
0.145. Similarly, all indicators related to agency and ownership of learning were at 
least marginally significant. The highest correlations were found in this category, with 
choosing to learn something new and giving feedback to peers showing the 
strongest relationships with project exposure. All but one of the collaboration 
indicators reached at least marginal statistical significance, as shown in Table 17. 


Table 17: Correlations Between Learning Studio Project Exposure and Focal Outcomes 


Agency & Ownership of 
Engagement & Persistence Learning Collaboration 
n=394 to 398 n=368 to 399 n=397 to 399 
Kept Working | 0.117" Took Risks | 0.097 Encouraged Peer | 0.123" 
Solve Multiple | 0.122 ° Set Own Goals | 0.080* Used Ideas From Peer | 0.105 
Stuck Tough | 0.145 Chose To Learn | 0.171 Worked With New Peer | 0.111 
Got Curious | 0.103 Gave Feedback | 0.180 Helped Brainstorm | 0.068 
Asked Questions | 0.087 Sought Feedback | 0.157 Helped Peer | 0.064* 


p<.001; p<.01; p<.05; *p<.10 


At the same time, with the exception of “Taking things apart” (r=.09*), no project 
exposure relation was found for design thinking and problem-solving outcomes. 
Though correlational and therefore not causal, this result suggests that students can 
benefit from intentional focus on aspects of design thinking in environments like the 
Learning Studio. “Taking things apart” was central to two project guides—“Parts, 
Purposes and Complexities” and “Toy Workshop”. In these projects, students chose 
an object to observe and carefully deconstruct in an effort to understand its purpose, 
function, and operating mechanisms. In the Toy Workshop project, students learned 
to create small, interconnecting parts and then use the parts to create toys or playful 
objects of their imagination. In contrast, for the other indicators of design thinking 
and problem solving, the number of projects completed did not seem to have an 
impact relative to other activities teachers implemented with students in the Learning 
Studio. 


For a second look at project implementation effects, we tested whether teachers’ 
responses to a selection of follow-up survey questions differed between those who 
had or had not chosen to implement the Play to Learn Challenge with their students. 
Results are presented Tables 18-19. For a variety of questions across the four focal 
outcomes, responses of teachers who implemented the Play to Learn Challenge 
were higher across the board, with the majority reaching statistical significance 
notwithstanding a small sample size. 
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YES 


NO 


Mean 


sd 


Mean 


sd 


Focal Outcome 


Through participation in the 
Learning Studio: 


Agency 


... |gained more comfort in 
leading projects in which 
students have a high degree 
of autonomy 


4.50 


HDS) 


4.26 


562 


t(25)=1.014 


Design 


... | grew in my ability to lead 
students In the design 
process 


4.63 


.518 


4.16 


.688 


t(25)=1.718" 


Skills 


... my students learned 
valuable new skills 


4.88 


554 


4.53 


ly 


t(25)=1.746° 


Empathy 


... my students improved in 
their perspective-taking ability 


.518 


4.26 


653 


t(25)=1.388" 


Collaboration 


... my students became better 
collaborators 


4.88 


554 


4.53 


513 


t(25)=1.746" 


Communication 


... my students improved in 
their ability to communicate 
their ideas to others 


4.88 


4.37 


t(25)=2.225" 


Empathy 


Taking part in the Learning 
Studios program helped my 
students to develop their 
knowledge and understanding 
of the world. 


4.38 


.518 


.667 


t(25)=1.416" 


Engagement 


Some students who are not 
typically engaged in classes 
really got into the Learning 
Studio projects. 


4.38 


.518 


4.26 


55) 


t(25)=0.39 


Engagement 


Students put in more time than 
was required on their Learning 
Studio projects 


4.13 


641 


809 


t(25)=0.713 


Agency 


The Learning Studio projects 
had personal meaning for 
students. 


4.63 


.518 


BAS) 


t(25)=1.487 
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Table 19: Play to Learn Challenge: Implementation Effect for Focal Indicators: Feedback 


YES NO 
Mean sd Mean sd 
Through the Learning Studio projects, 
students gained facility in: 
requesting feedback mes 7570 7o58)| t(25)=2.649" 
offering feedback Peeseese42nes0me)| ((24)=2.601" 
integrating feedback A 25) ||| 707 eee A71 | t(25)=1.083 


Teacher Background and Experience. The second set of analyses examined patterns 
in student outcomes in relation to teacher background and experience with 
technology and design thinking, using the groupings described in section 3.1 (See 
also Appendix B). The first group (Beginning), while evidencing enthusiasm and buy- 
in for the Learning Studio project, came with little background in making and did not 
indicate prior exposure to or awareness of design thinking principles or strategies. 
The second group of teachers (Emerging) described some personal experience with 
making, and generally did not bring experience facilitating students in making or 
design thinking. The third group (Intermediate) brought both personal and 
professional experiences with design, having facilitated student making projects, 
including robotics clubs and other initiatives. The fourth and final group of educators 
(Advanced) evidenced knowledge and experience of the design process, situating 
making within a larger culture of pedagogy related to project-based learning. Since 
students from teachers categorized in group 2 did not complete the post survey, the 
analyses compared groups 1, 3 and 4. 


To explore whether student outcomes were related to teacher groups, we carried 
out several one-way ANOVAs with LSD post-hoc pair-wise comparisons using 
Bonferroni corrections. Figure 29 provides a legend for reading the visualizations 
used to present the results of the group comparisons, which appear in Figures 30 
through 34. 
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GROUP 1 Connecting lines and 


y labels denote significance 


Student means and of pairwise comparisons 


(standard deviations) 
for each teacher group 


GROUP 3 : GROUP 4 
(Intermediate) (Advanced) 


Results of overall give classmates 
group mean feedback? 
comparisons ~~ F(2,283) = 5.007, p=0.007 
(ANOVA) 


For this question, we see that the average for Group 1 (2.9) was 
significantly lower than that for Group 3 (3.5). The difference 
between Group 3 and 4 was marginally significant; and the 
different between Group 1 and 4 was not significantly different. 


Figure 29: Legend and Guide for the Teacher Group Data Visualizations 


The first group comparison examined the extent to which students’ sense of identity 
as a designer or maker differed across teacher groups. As can be seen in Figure X, 
notable differences are present between group 1 relative to groups 3 and 4. For 
example, whereas at post-survey the average response to “Are you a Maker" for 
students of teachers in group 1 was 2.8, for students of teachers in groups 3 and 4 
the averages were 3.6 and 3.8, respectively. These differences were highly statistically 
significant. In contrast, statistically significant group differences were not detected for 
students’ overall ratings of confidence with making activities and processes (not 
illustrated). 


Are you a Maker? 
A2,334) = 16.584, p< 0.001 


Figure 30: Student Maker Identity x Teacher Groups 


We also explored patterns between students’ use and comfort with the technology 
and teacher background (Figures 31 and 32). We found that students’ Sprout use and 
comfort varied by teacher group, with students associated with group 1 having 
significantly lower use of the Sprout relative to students associated with groups 3 and 
4. Students’ comfort using the Sprout, and teaching others to use it, was highest for 
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those associated with group 4; however, the average responses for groups 1 and 3 
were Statistically equivalent. Regarding use and comfort with the 3D printer, average 
responses for students of group 4 teachers were higher across all four indicators 
relative to groups 1 and 3. 


‘ 
a 
g 

(3 ea) 


Used the Sprout? Comfort using Sprout? Comfort teaching Sprout? Comfort fixing Sprout? 
A2,333) = 1.931, p=0.147 A2,333) = 20.575, p<0.001 A2,334) = 7.86, p<0.001 A2,334) = 1.27, p=0.282 


Figure 31: Student Sprout Outcomes x Teacher Groups 
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Used the 3d Printer? Comfort using 3d Printer? Comfort teaching 3d Printer? Comfort fixing 3d Printer? 
F(2,333) = 3.806, p=0.023 A2,333) = 25.833, p<0.001 A2,334) = 18.709, p<0.001 A2,334) = 7.84, p<0.001 
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Figure 32: Student 3D Printer Outcomes x Teacher Groups 


Notably, students taught by educators in Group 1 had significantly lower Maker 
Mindset averages than did those taught by educators in Groups 3 and 4. Similar 
analyses were carried out to compare students’ post-survey responses for agency 
and initiative. Interestingly, students of teachers in group 3 posted the highest 
averages for these dimensions of Learning Studio experience, relative to other 
students. Figures 35 and 34 present the results. 


Communicating new Being open to new and Incorporating group input —_ Letting go of your idea in 
ideas to others diverse perspectives and feedback into work favor of an idea more 

F(2,245) = 6.579, p=0.002 F(2,245) = 13.75, p<0.001 F(2,245) = 10.575, p<0.001 supported by your team 
F(2,245) = 11.522, p<0.001 


Figure 33: Empathy, Communication and Collaboration x Teacher Groups 
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p =0.016 C35 eem( 28) 


give classmates ask classmates choose to learn something to set your own 
feedback? for feedback make your project better project goals 
F(2,283) = 5.007, p=0.007 F(2,285) = 11.306, p<0.001 F(2,284) = 3.735, p=0.025 F(2,284) = 6.982, p=0.001 


Figure 34: Student Agency and Ownership of Learning x Teacher Groups 


3.7 Program Reach 


In this final section on results, we present some additional findings regarding overall 
program reach, as well as teachers’ and students’ overall perceptions of the Learning 
Studio program. In total, this first implementation of the Learning Studio program 
reached well over four thousand students. Teachers reported that 2,286 students 
participated occasionally in the Learning Studio, 1,133 students participated regularly, 
and 805 were avid participants.” Among all participating students, teachers estimated 
that 326 were students with documented learning disabilities. 


Global Initiative. A hallmark of the Learning Studio program was the inclusion of 
international implementation sites, with opportunities for teachers and students to 
connect and even collaborate across classrooms. The follow-up survey included 
questions to teachers regarding the global nature of the program for their own 
professional development and program experience, as well as for their students’ 
learning. Overall, we found moderate support for this aspect of the program. The 
large majority (85%) of the 27 teachers who completed the follow-up survey agreed 
that taking part had helped their students to develop their knowledge and 
understanding of the world; half (52%) felt the international scope of the program had 
added value to their students’ experience. In terms of their own experience, more 
than three-fourths of teachers (77%) agreed that taking part in the Learning Studios 
global online professional learning community network had positively impacted their 
practice; 19% strongly endorsed this outcome. Seven in ten felt that their experience 
had been enhanced by the participation of teachers and students from other 
countries. Teachers’ ratings of the value of the online community were mixed, with 
61% expressing agreement, 31% remaining neutral, and 8% indicating disagreement. 
At the same time, there was some evidence in interviews that increasing cross-site 
collaboration was a desirable future goal for teachers. 


Ripple Effects. Teachers also described how their implementation of the Learning 
Studio had ripple effects throughout the school and district. A second-grade teacher 
explained, “| see it growing and spreading. We did a couple making activities with the 


* Teachers interpreted the labels “avid”, “regular” and “occasional” for themselves 
when responding to this question. 
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whole second grade and that started growing throughout the grades and now other 
classes, it’s started spreading in other grades. People are talking about paper circuits. 
It seems you start with a project or one classroom even and the kids can't help but be 
excited and tell other people about it and pretty soon lots of people are doing 
catapults and lots of people are doing little making projects. I've seen it really expand 
in my school. Now everyone is asking about 3D printing.” 


For one district in particular, the introduction of the UN Sustainable Development 
Goals in the Learning Studio led to a new culture. “In the very beginning [the Learning 
Studio team] said, we will base the projects on the UN Sustainable Development 
Goals. | brought that back to my team and now we use those Sustainable 
Development Goals throughout the whole district. We're always looking for real- 
world questions for kids, and by default those are amazing real-world questions. And 
the kids say, ‘It makes us feel that what we are doing is important. These are the 
things that the leaders of our world think we need to work on. And so we're working 
on them, too.’ That was the perfect reason to embrace that." 


4. Limitations 


As with all studies, several factors impact interpretation of findings and limit the 
extent to which the outcomes can be generalized to a broader population. One 
consideration in interpreting the outcomes reported here is whether the changes we 
observed can be attributed to the intervention itself, or whether other factors may 
have been responsible for the effects found. In situations where the program under 
examination is well defined and implemented in comparable contexts, changes can 
be confidently attributed to the program. These attributions are harder to justify 
when implementation differs substantially with respect to program definition and 
participant characteristics at each site. Because the Learning Studio was first and 
foremost a program offering, implementation decisions were driven by factors other 
than research design. This led to amplification rather than restriction of factors other 
than the Learning Studio that varied from site to site, and posed challenges to 
isolating key influences and causes. 


One limitation therefore pertains to the fact that we did not study a specific 
intervention, but rather a broad spectrum of choices at each site in how to utilize the 
technology and resources provided by the program. As seen in preceding sections, 
the open-ended nature of the Learning Studios, in which each teacher chose which 
projects to implement, if any, resulted in substantial variability in the nature and 
amount of student exposure. Moreover, these project choices interacted with an 
array of instructional contexts. In some cases, Learning Studios primarily enhanced 
the technology tools at students’ disposal within existing classes in entrepreneurship, 
or in engineering, such as Project Lead the Way. In other cases, maker learning 
programs were already underway and the Learning Studio contributed to ongoing 
expansion of that effort. And at some sites the Learning Studio introduced design 
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technologies for the first time, in contexts that had previously prioritized large-group 
direct instruction with students. Consequently, what was meant by “Learning Studio” 
was very different from site to site, with important implications for making sense of 
the data we gathered. For instance, it is possible that averages across the program 
were largely driven by gains in a couple classrooms; moreover, changes in those 
classrooms could have been due to processes already underway to drive student 
growth in design thinking. Sample sizes and large differences in number of students 
who responded from site to site constrained our ability to carry out meaningful class- 
level or teacher-level analyses. 


Lack of uniformity in participant characteristics is a further issue to weigh in 
interpreting the results. Teachers at each site differed in important ways, from their 
personal and pedagogical experience with design thinking and making to their 
experience and comfort with advanced technology. Moreover, grade levels spanned 
elementary through high school, and students had varied prior exposure to robotics, 
making, and engineering programs. When coupled with the differences in the 
Learning Studio program itself, this variability posed additional challenges to 
interpretation. In any study, the more that participants differ from one another within 
a variable intervention model, the harder it is to isolate the effects of the program and 
to obtain an accurate read on salient implementation features and contextual 
characteristics. 


Further considerations include response rates and instrumentation. As is typical of 
implementation research, we saw relatively low response rates, particularly for the 
follow-up survey, which limited our ability to identify trends over the full duration of 
the project. Moreover, many students who hadn't used the equipment responded, as 
revealed in students’ responses on the post survey. Also on the post survey, via open- 
ended responses a notable number (4%) of students indicated that they hadn't been 
to the Learning Studio or started the project yet. Results may have been diluted by 
the responses of students who didn’t experience the full implementation of the 
Learning Studio at their school. 


Finally, the research was limited by the lack of established measures and methods for 
studying making and design-based learning. The survey items we used may not have 
been sufficiently sensitive to the impacts that were caused by the Learning Studios. 
Perhaps more fundamentally, a potential limitation of the study was the decision to 
rely primarily on quantitative survey items to explore Learning Studio effects. This 
choice was driven in part by an interest in contributing to a base of evidence 
demonstrating the positive impacts of design thinking programs. In addition, the 
program design anticipated the specific outcomes for which we administered survey 
items. Because quantitative approaches are well suited to the participation of large 
groups, they are a good choice when the goal is to generate evidence regarding 
specific hypotheses about impact and the role of context. On the one hand, it could 
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be argued that what we found through survey data is that simply providing Learning 
Studio package is not universally sufficient to generate impacts. Another perspective 
is that it was too early in the arc of studying the benefits of design thinking programs 
to focus on hypothesis testing, and that a primarily qualitative approach may have 
afforded more opportunities to uncover trends and insights. 


5. Insights for Future Implementations 


As referenced in several places in this report, the Learning Studio was a highly open- 
ended program, and implementation at each site reflected a great deal of diversity. 
This variation is important to consider when interpreting results and drawing insights 
that could assist in future implementations of this and related programs. Overall, 
while we found evidence for a range of positive benefits associated with the Learning 
Studio program, these benefits were not universal across all students. Measurements 
of growth from pre to post, using both quantitative as well as qualitative items, 
yielded more conservative estimates of impact than did questions regarding students’ 
and teachers’ perceptions of growth at the end of the program. 


We were curious to understand the conditions in which students drew the most 
benefit from their Learning Studio experience. As described in Section 3.6, we found 
that one explanation for differences in student outcomes was teacher background, 
and another was the extent to which students used the Learning Studio project 
guides. From interviews, we noticed that differences in teachers’ prior experiences 
with designing and making translated into different kinds of classroom culture, which 
were potentially responsible in turn for differences in student outcomes. Table 20 
describes the general patterns that emerged from the overall data set. 


Table 20: General Patterns in Contextual Factors and Student Outcomes 


Teacher Teacher Comfort Instructional Student Student 
Background Focus Experience Outcomes 
- Low to 
moderate - Technology Use 
Limited design comfort - Production of - Procedural - Tech skills 
thinking and - Low to objects - Somewhat = Collaberstive sititudes 
PBL moderate - Teacher-defined collaborative 
comfort as projects 
facilitator 
- Tech skills 
Prior - High tech - Strengths-based Student-driven |~ Persistence 
SAPSHERGS comfort Student agency : Identity as 
with design ; are - Collaborative ; 
ae - High comfort as Identification of : Designer/Maker 
thinking and “i - |terative F ee 
PBL facilitator problems to solve Collaboration abilities 
Design thinking 


Overall, it appeared that the nature of program benefits depended on whether the 
context was an emergent or established Learning Studio environment. In what we 
could call “emergent” contexts, the focus tended to be on developing technology 
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skills and executing specific assignments. This could be seen in the ways that 
teachers at these sites described the implementation, and in students’ open-ended 
responses about what they had learned in the Learning Studio. In “emergent” settings, 
another benefit of the Learning Studio was to open new possibilities for the uses of 
technology as well as for the importance of fostering students’ non-cognitive skills 
such as collaboration, time management, persistence and problem-solving ability. In 
contrast, in “established” Learning Studio contexts, some elements of student-driven 
learning, design thinking and collaboration were in place prior to program 
implementation. In these settings, the Learning Studio enabled greater design focus 
and cross-site collaboration than previously experienced. While the outcomes at 
“established” sites were generally stronger, the Learning Studio was one of many 
causal contributors; in “emergent” sites, Learning Studios served a larger role as a 
catalyst for new ways of engaging students in 21°" century competencies. 


For example, at one “emergent” site, a major focus was the production of cookie 
cutters by all students at the school. In focus groups at this site, most students 
characterized their experience of the Learning Studio as fun but offered little 
evidence regarding growth in problem solving or design thinking. An educator from 
this site noted that the introduction of the Learning Studio at the school revealed the 
extent to which students’ prior classroom experiences had failed to develop their 
collaboration skills. He found that through the Global Goals, Local Solutions project, 
students came to realize the value of working with others to be successful. In 
contrast, at “established” sites, Learning Studio lead teachers had already created 
classroom environments in which students were actively encouraged to take risks, 
were provided with many opportunities for authentic group work and collaboration, 
and were focused on identifying meaningful problems to solve. For these teachers, 
the Learning Studio enabled them to enhance their students’ design thinking and 
collaboration skills. 


A related insight from the research is the apparent relationship between teachers’ 
own comfort and familiarity with technology and design-based learning, and the 
nature of the experiences they offered students. These differences were independent 
of a willingness to implement the Learning Studio, which all the teachers seemed to 
take on eagerly. For example, at an emergent site, the lead educator shared that 
rather than focus on student outcomes in this first implementation, his goal was to 
enable teachers to build their own comfort with this new paradigm for teaching and 
learning. In some cases, teachers’ own lack of comfort translated to reluctance to 
allow students to explore the technologies on their own. 


One perspective on the Learning Studio program is that it represents an effort to 
scale the “Maker Movement” and design-based thinking into formal learning 
environments. As Is true for any expansion of an initiative from organic growth to 
systemic adoption, the drivers of successful implementation may be different, and 
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may carry new implications for program design. Until recently, “makerspaces” were 
primarily born of an educator's personal passion for designing and making. It is likely 
that individuals drawn to the “do it yourself” and design communities share 
characteristics Such as an openness to risk-taking, comfort with students 
experimenting, and a commitment to interest-driven project work. Indeed, as shown 
in Appendix E, teachers in our study who indicated prior experience leading design 
thinking and makerspaces reported significantly higher levels of comfort across a 
number of the skills and abilities we surveyed. Moreover, teachers’ experience leading 
makerspaces was related to their persistence in the Learning Studio, as measured by 
completion of later surveys. Close to twice as many teachers with prior experience 
completed a post or follow-up survey (38%), compared to 22% of teachers without 
prior experience. 


When the idea of a makerspace or learning studio goes to scale, the same 
assumptions are unlikely to hold — whether for the educators or for the participants 
themselves, who are no longer volunteers but required. To the extent that robust 
outcomes are valued, this context of “scaled” organic implementation may demand a 
more intentional approach to developing design thinking, and moving educators and 
learners along a trajectory from technology skills and production focus toward a 
culture of maker mindset, design, and iterative problem solving. 


In light of these observations and the findings from this report, we identified four 
main areas of insights for future implementations. 


1. Time and Technology 

Comfort with technology grows over time. Making sure teachers have ample 
opportunity to familiarize themselves with the technology, and ensuring that the right 
infrastructure is in place, can be a simple way for teachers to build their personal 
comfort before being in the position to facilitate students’ use of it. In our study, 
many teachers noted that time was a critical factor, and expressed the desire that 
there had been more lead time—whether for getting the technology fully set up, for 
familiarizing themselves with the basics, or for introducing students to foundational 
knowledge and attitudes in a design-based learning paradigm. This latter mindset 
appeared more difficult to influence in older students relative to younger students — 
likely due to their longer-term exposure to expectations and mindsets of traditional 
instructional models over years of schooling. Related to time, teachers also noted the 
importance of full buy-in and expectation-setting at the school and district level, to 
ensure that the needed technology supports were in place for the software and 
hardware. 
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2. Professional Development and Mentorship 

Findings related to teachers’ facility with advanced technology, and especially with 
design thinking and problem-solving processes, suggest that an important focus for 
future implementations is extended professional development opportunities for 
teachers, both prior to and during implementation. This support could take a variety 
of forms. One option is for teachers to take part in hands-on experiences with the 
projects, ideally in a group setting, with an experienced educator serving as a model 
for facilitating the process. Many teachers are unfamiliar with common design 
thinking processes and human-centered design practices. As such, they are in the 
challenging position of being expected to discover best practices on their own, 
without the benefit of wisdom from those who have been through the process many 
times. While each teacher's learning will still reflect a personal journey, exposure to 
best practices can accelerate their learning curve. Findings from this study suggest 
that the more teachers have some personal experience and have developed their 
own mindsets reflective of design thinking, the more likely their students are to take 
on these practices as well. 


Teachers shared that additional mentorship could be beneficial. Although the 
Learning Studio offered an online discussion and sharing site, most teachers did not 
engage deeply with it. One teacher noted the difficulty of developing effective 
working relationships in the virtual large-group format, which can remain somewhat 
anonymous particularly when teachers have not met in person. One model that met 
with enthusiasm from teachers we talked to is a small professional learning 
community (PLC) of two advanced and two novice teachers, who share insights and 
tips. Some teachers were eager to increase the level of student-to-student 
collaboration across sites, which could also be accomplished within a small PLC. 


3. Intentional Instruction 

The role of pedagogy is another important consideration for future implementations. 
One perspective is that providing teachers and students with access to technologies 
and activity prompts will necessarily lead to substantive growth in the kinds of 
abilities described by the focal outcome areas, such as design thinking, empathy and 
student ownership of learning. Beyond improved technology skills and comfort, the 
results of this exploratory study do not bear that hypothesis out. Instead, the data 
suggest that intentional instruction is an important context for students’ development 
of design thinking and related competencies. This is especially likely to be the case 
when students have not self-selected into the Learning Studio experience, and may 
be more naturally drawn to interests other than designing and making activities. 


The teachers in our study who described the most impressive gains in students’ 
persistence, collaboration, and other skills were those with pedagogical practices 
specifically targeting these areas of growth. For these teachers, the Learning Studio 
program provided occasion to continue, enhance and deepen. At sites where a 
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design thinking culture was not already present, the nature of gains was more 
modest and concentrated in technology skills, with some progress in collaboration 
and communication. While traditional direct instruction would not be in keeping with 
a design thinking or maker philosophy, intentional instruction—in which students are 
facilitated to extract a design process that includes understanding the problem 
situation, identifying root causes, prototyping and testing—appears Key to fostering 
student growth. This insight is further demonstrated in the fact that the more project 
guides teachers used, the more students demonstrated growth in the focal 
outcomes. 


A related perspective on pedagogy is the value of setting expectations for students’ 
learning in advance. Expectations could range from completely open—including the 
possibility that students take little from the experience—to more specific, such as a 
focus on aspects of collaboration, or design thinking, or the development of 
empathy. By articulating expectations prior to implementation, educators can craft a 
Learning Studio experience that facilitates those expectations and ensures ample 
opportunity for students to grow in valued competencies. This is especially true of 
implementations in formal educational environments where there is a mandate to 
engage students in experiences that contribute to their learning and growth. 


4. Scope of Implementation 

Learning Studio sites differed in the scope of their implementation. In some cases, 
entire grade levels were involved; other programs focused on a small group of 
students. Figuring out the right grain-size for initial implementation Is an important 
consideration to weigh. Our observations suggest that starting on the smaller side of 
the spectrum, and ensuring that teachers have ample opportunity to grow their own 
confidence and comfort levels, lead to the greatest student gains. Future renditions 
of the Learning Studio might include recommendations for scope tied to the extent 
of prior design projects, as well as teachers’ own facility in a design-based learning 
environment. 


6. Recommendations for Future Research 


This exploratory study into the implementation and impacts of the Learning Studio 
program yielded a variety of insights, and gave rise to additional questions for future 
research to pursue. As the results presented in this report show, design-based 
learning experiences like those afforded by the Learning Studio offer promise and 
potential for student learning and for the development of the “non-cognitive” skills— 
such as persistence, collaboration and problem solving—that are increasingly 
recognized as important to life success. There is a need for more studies to provide 
compelling documentation of the value of these programs. Achieving this goal 
requires a research agenda comprising multiple studies to qualitatively gather more 
nuanced understandings of the role that context and implementation models play in 
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fostering teacher and student outcomes; subsequent large-scale research using 
quantitative methods are also needed to confirm and validate patterns. 


Among avenues for additional inquiry are the role of intentional instruction for 
promoting aspects of design thinking in students, and the best practices for growing 
teachers’ comfort and confidence in a design-based instructional paradigm. Another 
question that arose in interpreting results, and even in setting out the initial goals for 
the program, was determining a reasonable level of change to expect in teachers and 
students after only a few months of implementation, and with various levels of 
support. Future research could document typical trajectories of growth In design 
thinking, including the amount of time it takes to develop at different grade levels, 
and the characteristics of instruction that are most conducive to students’ 
engagement as well as learning. 


As outlined in earlier sections of this report, isolating the effects of a specific program 
or contextual factor is difficult to do when the program is broadly defined, and 
participation contexts vary greatly. A related factor is the choice of method for the 
research. Among opportunities for future studies is the revision and validation of 
survey instruments to ensure that teachers and students are interpreting items as 
intended, and that response scales offer a range of responses sensitive to changes in 
attitudes, beliefs and competencies. Given the known issues with self-report data, the 
development of additional performance tasks for technology, collaboration, design 
thinking and problem solving would represent an important contribution to the field. 


7. Conclusion 


This paper presented findings from exploratory research of a one-year Learning 
Studio implementation at diverse sites in the United States and abroad. Results from 
quantitative and qualitative analyses suggest that many participating teachers and 
students grew in several aspects of advanced technologies and design-based 
thinking, and that access to the Sprout and 3D printing technologies opened new 
opportunities for students’ creativity and iterative problem solving. Findings also 
indicate that teachers’ background and students’ design-based behaviors are related, 
and that higher levels of teacher comfort with design thinking processes and 
technologies are predictive of students’ growth. Importantly, the evidence gathered 
through this research shows the potential of projects like the Learning Studio to have 
positive impacts on key skills and beliefs for students of the 21 century. We hope 
that the insights and questions raised in this report will contribute to ongoing efforts 
to bring opportunities like the Learning Studio to all children. 
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APPENDIX A 


Sample Project Guides from the Learning Studios Teacher's Guide 


Custom Cookie Cutter 


Make your own cookie cutter — choose a design that you like and think that you will 
be able to model. Teachers Guide | Student Guide 


Toy Workshop 


Make a toy! Now that you've had some experience with interconnecting pieces, build 
your own toy using stiff joints, loose joints and ball joints. Teachers Guide | Student 
Guide 


Global Goals, Local Solutions Design Challenge 


Participating sites around the world are invited to take part in a culminating 2016 
Learning Studio Global Goals, Local Solutions Design Challenge. Entries can take any 
form — from devices to media campaigns to processes - anything students can 
imagine that incorporates their creativity, designs, and skills to address real-world 
issues that matter to them. Challenge Guide 


Play to Learn Design Challenge 


How might we create a digital or analog game to help the player(s) learn something 
new? Teachers Guide | Student Guide 
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APPENDIX B 


Selections of Teacher Pre-Survey Responses Representing Each Grouping 


Ways | expect to change 
personally 


How will students 
grow? 


What will students learn? 


with making, 


Group 1: US | feel that the Learning We hope they grow | hope students will learn 
Beginner Studio will truly push my | and expand their to be innovators and 
(enthusiastic classroom into the 21° innovative thought enjoy learning. In 

7 exe 

but new to century and have a place to addition, learning goes 
maker apply them and not beyond textbooks. 
learning) just dream. 

US | love to learn about new | Locating the Learning | Creating projects can be 
technology and ways to__| Studio in the library [an] excellent way to 

678 keep students engaged will be an additional blend/apply skills from a 

way to keep students variety of curriculum 
engaged in the many areas. 
activities/events we 

have throughout the 

year. 

US | expect to become | think they will grow | hope they learn how to 
more comfortable with with their confidence. | collaborate and think 

KS incorporating creatively. 
technology into the 
classroom setting. 

Int Change the style and Get connected to and | Development of IT 
methods used in the share ideas with other | competencies, social 
classroom at the Learning Studio competencies, 

6 10 : 

12 moment students of different improvement of . 

parts of the world. presentation techniques. 
Group 2: US New experiences and In creativity and How to learn to ask the 
Emerging learnings willingness to learn right questions to solve 
(some 11 from mistakes problems 
personal 12 
experience 
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but new to 
design 
process) 

Int | expect learning a lot They will take New ways of learning, in 
about new methods of advantage of this collaboration with 
teaching and new ways project as a way to others, involving the 

10- ; é tae 

12 of interaction with my cooperate with others entire community in the 
students. and I’m sure that they | [creation] of knowledge 

will progress a lot in 

their development as 

future adults. 
Group 3: US Shifting more to | hope they will Creative thinking and 
Intermediate student-centered develop a love for independence with 
(focus and 5 learning even more so building and computer | technology. 
experience than | already use. sciences. 
with students’ 
making; 
design 
thinking not 
explicit) 

US | will enjoy learning to Students should How to self-regulate. 
use 3D technology and become more How to organize a 

11 reacquainting myself empowered to take group to maximize 

1D with CAD. | will also control of their own productivity by 
enjoy learning how to learning because they | leveraging strengths (...). 
introduce project-based | can see how it is | hope students will learn 
learning into my relevant to practical how to use the current 
curriculum. applications in which technology in the 

they have an interest. learning studio to 
manifest their creative 
ideas. 

US | hope to transform my | hope they will grow That school is more than 
library from a place of with each other and just answering questions 

9-12 only consumption of the collaboration on a test or writing an 
information to a creation | teaches them how to __ | essay. That inquiry and 
studio. | want students work as a team, ask problem solving and 
to talk, create, dream, for help, use creating is important. | 
share, and innovate resources, find hope they learn that 
together. resources, and to not | technology is a powerful 

be afraid to change tool that can enable 
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the world! 


them to share their gifts 
but innovation comes 
from within. 


Int More confident with Develop skills to A How to integrate new 
7-12 | new technology level .. build on these technology to support 
and transfer across and enhance problem 
other subjects. solving/modelling and 
realisation of ideas. 
Group 4: Int Comfortable with the Form new friendships; | Understand the 
Advanced new technology; develop interpersonal | technology; improved 
(background 456 integration of learning Skills; improve collaborative learning; 
in project- areas to still cover communication skills new insights into 
based required concepts in science/math/technolog 
learning; math/sciences/technolo y concepts; a better 
awareness of gy understanding of the 
design design and marketing 
process) process in the real 
world; cooperative 
communication skills 
US | expect people to get | hope students learn | hope students will fall 
excited about the maker | how to bea in love with learning. | 
> movement and 21° successful global hope students will 


century learning. | 
expect people to seek 
further knowledge about 
Learning Studio and find 
ways to benefit from it. 


learner. 


develop a growth 
mindset. | hope students 
will learn how to 
persevere (how to fail 
but recover and 
succeed). | hope 
students learn to be 
thinkers and problem 
solvers. | hope students 
learn how to be 
producers of their 
dreams. 
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| think | will be growing 
as an educator to move 
more from 
demonstrating and 
lecturing to a facilitation 
role and allowing 
students to be more “in 
charge” of their own 
learning. 


| want them to learn 
the technology skills 
inherent in using the 
Sprout, and the 3D 
printer, but | think the 
biggest thing | want 
them to come away 
with is the change in 
thinking—to be more 
independent, to 
question, to believe 
that they can make 
change. 


| hope students will learn 
to use the design 
thinking models, and 
become creative risk- 
takers. 
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APPENDIX C 


Students’ Comfort with Learning Studios Equipment — Without removing students 
who had used the Sprout or 3D printer at least once by POST or FOL* 


PRE POST FOL PRE-POST PRE-POST-FOL 
Mean| Sd |Mean| sd Mean sd 
- Use 2.92 |1.480| 3.23 | 1.367 t(400)=4.208 
2.40 |1.450] 2.72 |1.469| 2.95 | 1.54 F(2,84)=3.254 
2 
- Teach 2.22 |1.329| 2.60 | 1.336 t(400)=5.283 
1.88 |1313| 212 [1.331] 2.63 | 1.61 F(1.683,70.695)=4.930° 
9 
- Fix 2.18 |1.248) 2.39 | 1.242 t(400)=3.335 
1.84 |1.067| 198 [1144] 2.56 | 151 F(1.743,73.223)=7.534 
7 
3D printer 
- Use 2.99 |1.469| 3.23 |1.368 t(400)=3.178 
2.42 |1.468| 2.63 11363] 2.74 | 1.43 F(2,84)=1.021 (ns) 
2 
- Teach 2.25 11.364] 2.59 | 1.394 t(400)=4.558 
Pomme eam 2020406) 2.4 1.51 F(1.616,67.859)=2.067 (ns) 
4 
- Fix 2.12 11197] 2.24 | 1.243 t(400)=1.796 
1.88 |1159| 1.74 /1.026] 2.35 | 1.46 F(1.557,65.396)=4.308° 
2 


" 5-point response scale ranged from 1 (“not at all comfortable”) to 5("totally comfortable’). 


“p<.01, “p<.05, ns = not significant 
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APPENDIX D 


Overall and Grade-Level Pre-Post Changes in Student Comfort with Communication 
and Collaboration Indicators 


PRE POST FOL 
Mean sd Mea sd Mean sd PRE-POST PRE-POST-FOL 
n 

Communicating new BOcmimecetm| 3.07 | 878 t(304)=.066 (ns) 
ideas to others 

3.26 | 795 | 3.08 | .784 | 3.34 | .745 F(2, 73.89) = 2.337 
Being open to new and Ceicmaimecoom| 35.21 | 825 t(304)=1.379 (ns) 
diverse perspectives 

3.47 | 060 | 3.42 | 642 | 3.50 | .558 F(1.94, 71.68) = 0.214 

5 

Incorporating group input | 3.10 | .786 | 3.06 | 821 t(304)=.830 (ns) 
and feedback into work 

Srigeicosm| 35.18 652 | 3.45 | .645 F(1.98, 73.25) = 2.356 
Letting go of your idea in Colieeeoem)| 5.01 | 881 t(304)=-1.750 (ns) 
favor of an idea more 
supported by members of 
your team 

S21 HUY Seismieeooom| 3.50 | 647 F(1.94, 71.79) = 4.209* 
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APPENDIX E 


Teachers’ Comfort for Those with and without Previous Makerspace experience 


Previously led maker space or design No (n=22- Yes (n=37- 
thinking programs? 23) 39) 
PERSONAL COMFORT peel eq 1) eee || sad Ie 
n n Comparison 
Using a Sprout computer 1.74 .964 2.56 | 1.088 t(60)=2.258* 
Using a 3D printer 1.78 902 | 2.62 | 1.042 t(60)=3.190* 
Taking something apart to see how it 270 | 926 | 3.46 | 0.790 t(60)=3.459** 
works 
drawings and/or models to share an 309 | 900 | 367 4 t(60)=3.200** 
Recording and editing an audio or video 3.00 1.04 356 0.68 t(33.185)=2.316* 
recording 4 0 
Creating a digital 3D model of an object 1.91 .868 2.67 | 1.009 t(59)=2.957** 
Assembling something WITH instructions _ 4 
(ex IKEA furniture or a Lego kit) 3.36 | 848 | 3.79 | 0.522 t(30.185)=2.166 
Assembling something WITHOUT 235 | 992 | 3.33 0.80 t(60)=4.287*** 
instructions 6 
Defining problems to investigate 3.22 671 3.62 ee t(60)=2.437** 
Working effectively when process is 3.09 596 351 0.64 t(60)=2.584** 
variable or ambiguous 4 
Communicating new ideas to others 3.74 541 3.79 —- t(60)=0.459 
Belg open to new and diverse 3.87 | 344 | 3.97 | 0160] t(27.708)=1.374+ 
perspectives 
Incorporating group input and feedback 365 573 3.72 0.45 t(60)=0.498 
into work 6 
Letting go of your idea in favor of an idea 3.26 541 358 | 0.552 t(59)=2.198* 
more supported by members of your team 
No (n=22- Yes (n=37- 
23) 39) 
COMFORT FACILITATING STUDENTS mean| sd | S* | sq ecm 
n Comparison 
Using a Sprout computer 2.05 | 1.133 | 2.34 | 1.072 t(58)=1.012 
Using a 3D printer 2.14 | 1125 | 2.63 | 1125 t(58)=1.643+ 
Taking something apart to see how it 281 680 | 3.26 0.86 t(49.927)=2.228* 
works 0 
es drawings-endrormodetstarshate:2t> |) 335°.) yao | see |0-474.| 140.047)=2.000* 
Recording and editing an audio or video 314 889 351 | 0731 t(57)=1.767* 
recording 
Creating a digital 3D model of an object 2.14 834 2.57 1.119 t(53.97)=1.686* 
Assembling something WITH instructions z : 
(ex IKEA furniture or a Lego kit) 3.45 739 | 3.79 | 0.528 t(33.598)=1.869 
Assembling something WITHOUT 245 858 321 0.84 t(58)=3.325*** 
instructions 3 
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Defining problems to investigate 3.41 590 | 3.55 5 t(58)=0.856 
Working effectively when process is 3.27 456 | 3.34 0.70 t(57.247)=0.461 
variable or ambiguous 8 

Communicating new ideas to others 3.82 | .395 | 3.68 | 0.574 t(56.123)=1.067 
Being Open to neWAlchaivelss 3.91 | 294 | 3.79 | 0.474] — t(57.65)=1.205 
perspectives 

Incorporating group input and feedback 373 | 456 | 368 | 0.525 t(58)=0.321 
into work 

Letting go of your idea in favor of an idea 359 | 590 | 355 | 0.555 (58) =0.252 


more supported by members of your team 
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